GEOMETRIC QUANTIZATION FOR PROPER MOMENT MAPS 



XIAONAN MA AND WEIPING ZHANG 

Abstract. We establish a geometric quantization formula for Hamiltonian actions 
of a compact Lie group acting on a non-compact symplectic manifold such that the 
associated moment map is proper. In particular, we resolve the conjecture of Vergne in 
this non-compact setting. 



0. Introduction 

The famous geometric quantization conjecture of Guillemin and Sternberg [9] states 
that for a compact pre-quantizable symplectic manifold admitting a Hamiltonian action 
of a compact Lie group, the principle of "quantization commutes with reduction" holds. 
This conjecture was first proved independently by Meinrenken [H] and Vergne [23] for 
the case where the Lie group is abelian, and then by Meinrenken [15J in the general case. 
The singular reduction case was proved by Meinrenken- Sjamaar in [16J. There are also 
an analytic approach to the original conjecture developed by Tian and Zhang [20] as 
well as a proof developed by Paradan [17] by making use of the theory of transversally 
elliptic operators, see also [24J for an excellent survey. 

It is natural to consider the generalizations of the above results to actions on non- 
compact spaces. One of the aspects of this issue has been considered by Weitsman in 
[26], where the properness of the associated moment map is assumed. In her ICM2006 
Plenary lecture [25], Vergne made a quantization conjecture (under the assumption that 
the zero point set of the vector field generated by the moment map is compact), which 
generalizes the original Guillemin-Sternberg conjecture to this non-compact setting. A 
special case of this conjecture had indeed been verified already by Paradan in [18] where 
he proved a quantization formula valid for the case where a maximal compact subgroup 
of a non-compact real semi-simple Lie group acts on the co-adjoint orbits of the real 
semi-simple Lie group itself. 

The purpose of this paper is to establish a general quantization formula in this frame- 
work of a compact group acting on a non-compact space with proper moment map. As 
we will see, our result could be viewed as an extended version of the conjecture of Vergne, 
in the sense that we do not make any extra assumptions beside the properness of the 
moment map. 

To be more precise, let (M, u) be a non-compact symplectic manifold with symplectic 
form u. We assume that (M,uj) is prequantizable, that is, there exists a complex line 
bundle L (called a prequantized line bundle) carrying a Hermitian metric h L and a 
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Hermitian connection V L such that 

(o.i) 2 sV)'-'- 

We also assume that there exists an almost complex structure J on TM such that 
(0.2) g™{u, v) = u(u, Jv), u, v ETM 

defines a Riemannian metric on TM. 

Let G be a compact connected Lie group with Lie algebra denoted by g. We assume 
that G acts on the left on M and this action can be lifted to an action on L. Moreover, 
we assume that G preserves g™, J, h L and V L . 

For any K G g, let K M be the vector field generated by K over M. 

Let [A : M — > g* be defined by the Kostant formula [10] 

(0.3) 27T V /Z T/i(^) := //W = V£ M - L*, KG 0. 

Then /i is the corresponding moment map, i.e. for any K G g, 
(0.4) d/i(lT) = i k mu. 

We call the G action with a moment map \i : M — > g* verifying (10.41) a Hamiltonian 
action. 

From now on, we assume that the following fundamental assumption holds. 

Fundamental Assumption. The moment map fi : M — > g* is proper, in the sense 
that the inverse image of a compact subset is compact. 

Let T be a maximal torus of G, Cq C q* be a Weyl chamber associated to T, A* C g* 
be the weight lattice, and G = A* n C<3 be the set of dominate weights. Then the ring of 
characters R(G) of G has a Z-basis V 7 G , 7 G G : is the irreducible G-representation 
with highest weight 7. 

Take any 7 G G. If 7 is a regular value of the moment map n, then one can construct 
the Marsden-Weinstein symplectic reduction (M 7 ,c<j 7 ), where M 7 = /i _1 (G • 7)/G is a 
compact (as /x is proper) orbifold. Moreover, the line bundle L (resp. the almost complex 
structure J) induces a prequantized line bundle L 7 (resp. an almost complex structure 
J y ) over (M 7 ,c<j 7 ). One can then construct the associated Spin c -Dirac operator (twisted 
by L 7 ), D+ 1 : fi°' even (M 7 ,L 7 ) -> fi°' odd (M 7 , L 7 ) (cf. Section dH (IQIl ) on M 7 whose 
index 

(0.5) Q(L 7 ) := dimKer (p L + ^ - dimCoker (l>+ 7 ) G Z, 

is well-defined. If 7 G G is not a regular value of /1, then by proceeding as in [16J (cf. 
[TTI §7.4]), one still gets a well-defined quantization number Q(L^) extending the above 
definition^ 



On the other hand, let g* be equipped with an Ad^- invariant metric. Set TL = \/i 
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Then since fi is proper, for any c > 0, U c := 7i 1 ([0,c]) = {x G M : Ti{x) ^ c} is a 
compact subset of M. 



1 See also Section [Ol for a standard perturbative definition. 
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Recall that by Sard's theorem, the set of critical values of the function Tt : M — > R is 
of measure zero. 

Let X n = —J(d7i)* be the Hamiltonian vector field associated to 7i. 

For any regular value c > of 7i, one knows X H is nowhere zero on dU c = 7i _1 (c). 
Thus, according to Atiyah [1, §1, §3] and Paradan [TTJ §3] (cf. also Vergne [23]), the 
triple (U C ,X H ,L) defines a transversally elliptic symbol (corresponding to the Spin c - 
Dirac operator (twisted by L) on M ) associated to the G- action on U c . And according to 
Atiyah [Tj §1], it admits a well-defined transversal index whose character is a distribution 
onG. 

For any 7 G G, let Q(L)J G Z denote the 7-component of this transversal index. 

Theorem 0.1. For any 7 G G, there exists c 7 > such that Q(L)J G Z does not depend 
on c ^ c 7; with c a regular value ofTi. And Q(L)J =0 G Z does not depend on c > 0, with 
c a regular value ofTL. 

According to Theorem I0.1[ for any 7 G G, we have a well-defined integer Q(L)J not 
depending on the regular value c^> 0. From now on we denote it by Q{L)" f . 
We can now state our main result as follows. 

Theorem 0.2. For any 7 G G, the following identity holds, 

(0.6) Q(Ly = g(L 7 ). 

Remark 0.3. If the zero set of X n is compact, then Theorem 10. II was already known in 
[T7] and [25J, while Theorem 10.21 was conjectured by Vergne in [251 §4.3]. Thus Theorem 
10.21 can be thought of as an extended version of the Vergne conjecture. 

If we set 

(0.7) Q G (L)-°° = 0Q(L) 7 V; G , 

then by Theorem 10.21 Qg{L)~°° equals to the formal geometric quantization in the sense 
of [261 Definition 4.1] and [TH Definition 1.2]. In particular, it verifies the functorial 
quantization property described as follows. 

It is clear that if M is compact, then Theorem 10. II holds tautologically and Theorem 
10.21 is the Guillemin-Sternberg conjecture proved in [16]. Let (N,uj n ) be such a pair 
with N being compact, F the notation for the prequantized line bundle over N, etc. 
Combining Theorem 10.21 with the result [261 Theorem 1] (cf. also [T9l Theorem 1.5]) one 
gets the following functorial quantization result. 

Let L®F be the prequantized line bundle over M x N obtained by the tensor product 
of the natural liftings of L and F to M x N. 

Theorem 0.4. For the induced action ofG on (M x N, u>Q)u N ) and L®F , the following 
identity holds, 

(0.8) Q ((L®F) 7=0 ) = ^Q(LrQ(F)- 7 - 

7£G 
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By taking iV to be the orbits of the co-adjoint action of G on g*, one recovers Theorem 
10.21 from Theorem 10.41 immediately. Thus Theorem 10.21 and 10.41 are actually equivalent. 
In this paper we will write out our proof for Theorem 10.41 directly. In this way we also 
get a new proof of the following functorial quantization result due to Weitsman [261 
Theorem 1] and Paradan [191 Theorem 1.5], without using the symplectic cut techniques 
there. 

Corollary 0.5. Under the assumptions in Theorem \0.J\ the following identity holds, 
(0.9) Q ((L®F) 7=0 ) =Y J Q (L,) Q (F_ 7 ) . 

Now let K be a compact subgroup of G such that the moment map of the induced 
Hamiltonian action of K on M also verifies the fundamental assumption of being proper. 
Then by combining Theorem 11.21 ( 10. 7ft with [191 Theorem 1.3], one gets the following 
consequence on the relation between Qg(L)~°° and Qk(L)~°°. 

Corollary 0.6. Any irreducible representation of K has a finite multiplicity in Qg{L)~°° . 
Moreover, the following identity holds, when both sides are viewed as virtual representa- 
tion spaces of K , 

(o.io) q g (l)-°°\ k = q k (l)-°°. 

Our proof of Theorems 10.11 and 10.41 is analytic. It makes use of Braverman's analytic 
interpretation [5l §14] of the transversal index of a wide class of transversally elliptic 
operators covering the ones mentioned above. The main idea is that through the analytic 
interpretation of Braverman, one can further express this transversal index by using the 
Atiyah-Patodi-Singer type index@ for Dirac type operators on manifolds with boundary 
[2]. One can then apply the analytic methods developed in [20] and [22] to study the 
corresponding quantization problem. 

Indeed, after we interpret the transversal index by the APS type index, it is almost 
direct to prove Theorem 10.11 by applying the analytic techniques in [20] and [22] . 

The proof of Theorem 10.41 needs more effort. Tautologically, one would adapt the 
idea of two steps deformations appeared in [HI §3] to the current situation. The main 
point comes from the fact that, after choosing the suitable (already non-trivial) first 
step deformation, in order to pass from the boundaries of submanifolds of M x N to 
boundaries of specific forms obtained by the product of a boundary in M with N, one 
may encounter a lot of zero points of the vector fields used in the deformation. It is then 
necessary to eliminate the potential contributions caused by these possible zero points^ 

Recall that in the analytic proof of the original Guillemin-Sternberg conjecture de- 
veloped in [20], one already encounters the local pointwise estimates around the zero 
points of the Hamiltonian vector fields used in the deformation there. It is remarkable 

2 We will briefly call it as APS index in what follows. 

3 The situation considered in [18] is much simpler as there is no zero point of the vector fields used in 
the corresponding deformation, he can then apply directly the homotopy invariance of the transversal 
index. 
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that such kind of local estimates still hold in the current (apparently more sophisticated) 
non-compact situation. 

We would like to point out that one may indeed formulate Theorems 10.11 10.21 and 
10.41 directly using the APS type index which is more intrinsic (as the definition of the 
transversal index involved depends on certain extra structures which go beyond the 
symplectic structures involved). 

It would also be interesting to give a direct analytic proof of Corollary 10.61 without 
using the symplectic cut techniques in [19] . 

The rest of this paper is organized as follows. In Section [TJ we first apply the results 
of Braverman [5] to interpret certain transversal index as a kind of APS type index. We 
then apply the analytic approach of the quantization formulas developed in [20], [22] 
and prove Theorem 10.11 In Section [2} we present our proof of Theorem 10.41 modulo a 
vanishing result, Theorem 12.41 whose proof will be carried out in Section [3j 

Some results of this paper have been announced in [13] . 

Acknowledgements. We would like to thank Professor Jean-Michel Bismut for many 
helpful discussions. The work of the second author was partially supported by MOEC 
and NNSFC. Part of the paper was written while the author was visiting the School of 
Mathematics of Fudan University during November and December of 2008. He would 
like to thank Professor Jiaxing Hong and other members of the School for hospitality. 

1. Transversal index and quantization for proper moment maps 

In this section, we prove Theorem 10.11 In doing so, we first express the transversal 
index appearing in the context as certain APS type index, with the help of a result 
by Braverman [5], then we apply the analytic methods developed in [20J and [22] to 
complete the proof of Theorem 10.11 

This section is organized as follows. In Section 11.11 we recall the definition certain 
transversal index in the sense of Atiyah [1] and Paradan [17] for group actions on man- 
ifolds with boundary. In Section 11.21 we consider certain APS type index in the same 
framework as in Section 11.11 In Section 11.31 we prove an invertibility result for some 
induced boundary operator. In Section 11.41 we introduce the idea of spectral flow in our 
context and use it to complete the proof of a result stated in Section 11.21 In Section 11.51 
we give an APS index interpretation of the transversal index in Section [1J] by applying 
a result of Braverman [5]. In Section IT761 we prove Theorem 10. II by applying the analytic 
method developed in [20] and [22 j . 

1.1. Transversal index. Let M be an even dimensional compact oriented Spin c -manifold 
with non-empty boundary dM . Let E be a complex vector bundle over M. 

Let G be a compact connected Lie group with Lie algebra denoted by g. We assume 
that G acts on the left on M and that this action can be lifted to an action of G on E. 

Let the tangent vector bundle tc : TM —>■ M carry a G-invariant metric g™ . Then 
one identifies TM and T*M via g™ . 

For K G g, we denote by K^f = -j^e~ tK x\ t =o the corresponding vector field on M. 
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Following [H p. 7] (cf. PH §3]), set 

(1.1) T G M = {veTM : (v(x), K M {x)) = for all Keg}. 

Let S(TM) = S + {TM) © S_(TM) be the bundle of spinors associated to the spin c - 
structure on TM and g™ . For any V G TM, the Clifford action c(V) exchanges 
S±(TM), the Z 2 -grading on S(TM). 

Let \I/ : M — > g be a G-equivariant map. Let ^ denote the vector field over M such 
that ty M (x) equals to the value at x of the vector field generated by ^>(x) G Q over M. 

We make the assumption that \1/ M is nowhere zero on dM. 

Let o$ )9 : Hom(vr* (5+(TM) ® E), ir*(S-(TM) ® E)) denote the symbol defined by 

(1.2) a^(x, v(x)) = vr* (V^Tc(v + V M ) ® Ids) | (a . ^ for (x, G T.M. 

Since ^ is nowhere zero on dM, by (11.11) and (11.21) one sees that the zero set of the 
restriction of cr|f^ on TqM is contained in a compact subset of M C TqM (where 
M = M\ dM is the interior of M) . Thus, it defines a G-transversally elliptic symbol on 
T G M in the sense of Atiyah pQ §1, §3] and Paradan [T71 §3] (see also Vergne |23j), which 
in turn determines a transversal indexB 

(1.3) ind (4y = ind 7 (4y • Vf, 

7GG 

with each Ind 7 (cr|f^) G Z. 

1.2. The Atiyah-Patodi-Singer (APS) index. We continue the setting in Section 

o 

Let h E be a G-invariant Hermitian metric on E, V E a G-invariant Hermitian connec- 
tion with respect to h E . Let }i s \™)® E be the metric on S(TM) <8> E induced by the 
metrics on S(TM) and on E. 

Let V 5 *™-* be the Clifford connection on S(TM) induced by the Levi-Civita con- 
nection V™ of g™ and a Hermitian connection on the line bundle defining the spin c 
structure. Let ^s{tm)<»e ^ e ^ e Hermitian connection on S(TM) <g> E obtained by the 
tensor product of the connections V S< - TA/ ^ and V £ . 

Let dv M be the Riemannian volume form on (M, g™). For s G tf°°(M, S{TM) ® E), 
its L 2 -norm ||s|| is defined by 

(1-4) II « Mo = / \s\s(TM)®E( x ) dv M(x). 

Jm 

Let (■, ■) be the inner product on ^ 7oo (M, S(TM) ® E) corresponding to || • ||q. 

One can construct now the Spin c -Dirac operator (twisted by E) (cf. [TT], Appendix 
D])by 

dimM 

(1.5) D E = c(e i )Vf i (TM)0i? : ^°°(M, S{TM) <g> E) -> ^°°(M, S(TM) ® £?), 
i=i 

where {ej} is an orthonormal basis of TM. Then D E is G-equivariant and formally 
self-adjoint. Denote by D E the restrictions of D E on ^^(M, S±(TM) ® E) respectively. 

4 We will not exploit the distribution nature of this index here. 
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Following [8, p. 139], let e > be less than the inject ivity radius of g™. We use 
the inward geodesic flow to identify a neighborhood of the boundary dM with the collar 
dM x [0, e\. Let e d i m Af be the inward unit normal vector field perpendicular to dM. Let 
ei, • • • , edimAf-i be an oriented orthonormal basis of T<9M so that e\, ■ ■ • , e^mM-i, ^dimM 
is an oriented orthonormal basis of TM\qm- By using parallel transport with respect 
to V™ along the unit speed geodesies perpendicular to dM, e\, • - • ,edi m M forms an 
oriented orthonormal basis of TM over dM x [0,e]. 

Let D§ M : tf°°(dM, (S(TM) ® E)\ aM ) -> %?°°(<9M, (5(TM) ® £)| 8M ) be the induced 
(by _D E ) Dirac operator on dM defined by (cf. [8, p. 142]) 

dimM-l dimM-1 

(1-6) Dg u = - Yl c{e^ M )c{e i )V s e ^ E + l - E tt«, 

i=l i=l 

where 

(1-7) 7Tjj := <V™e i; e dimA /> | aM , 1 ^ i, j < dimM - 1, 

is the second fundamental form of the isometric embedding iqm '■ dM <^-> M. Let D§ M ± 
be the restriction of L>f M to ^°°(dM, {S±{TM) ® E)| 9M ). 

As in ( ll.4l) . we define the inner product {-,-)q M q anc ^ ^he -^2-no rm || • ||aA/,o on 
^^(dM, (S(TM) E)\q M )- 

By Lemma 2.2} D§ M is a formally self-adjoint first order elliptic operator intrinsi- 
cally defined on dMu 

On the other hand, since G acts on M and thus preserves <9M, one has 

(1.12) y M \ 9M G ^°°(dM,TdM). 



5 In fact, one has for any s £ <tf°°(M, S(TM) <g> £7), the following identities hold on dM, 

( 1 dimM-l \ 
^Im"2 E ^JWa^). 

C (e d imAf) £>g M (s|©Af) = —Dg M c(e d imA/) {s\dAl) ■ 

In particular, let \7 TdM be the Levi-Civita connection on (TdM, g TdM ), with g TdM the metric on TdM 
induced by g™ , wc define c(AT) = — c (edimAf) c(AT), for any X £ TdM, then c(-) induces a Clifford 
action of the Clifford algebra C(TdM) on (S(TM) <g> E)\qm, an d the Hcrmitian connection 



dimM-l 

(1.9) V^ v : =V^ ; +- 2^ TyC (edimAf) ^ejO^ei), 



is the c(.)-Clifford connection on (S(TM) ® -E)|dAf, verifying for example that for any A", V e 
■r°°(aM, TdM), 



(1.10) v ^(TAf)®E)| eM ~ (y) 



= ?(v™ M y). 



Thus Df M is the Dirac operator on (S(TM) <g> £7)| 9M associated to V (s,(TM) ® E)leM , i.e. 

dim A/-1 

(1.11) Df M = ^ ?(e,)Vif T W)l- 



3=1 

'3M 1 



From p. lip , we understand that Dg M is formally self-adjoint and intrinsically defined on dM. 
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By ( TTT21 and following [SI Lemma 2.2] and [22, §lc)] (see also §2.8]), set for T G 
;i.!3) 



D% = D E + - 2 1T c (* M ) : (M, 5(TM) ® P) -> (M, S(TM) ® E) , 
Df >T = Pf + ^y^c (^ M ) : (M, S±{TM) g> P) -> <af°° (M, 5 T (TM) g> P) , 

and 

^Im,t = ^Im - c (e dimA f) c (vf M ) 

( L14 ) : (<9M, (S(TM) ® P) | aM ) -> (5(TM) ® P) | aM ) , 

D dM,±,T = D dM,TW°°(dM,(S ± (TM)®E)\ aM )- 

Then Pf MT is also formally self-adjoint and preserves (<9M, (^(TM) ® P) |sm)- 
For any A G Spec{Pg M ± T }, the spectrum of Dq M± t , let Pa,±,t be the correspond- 
ing eigenspace. Let P^q,±,t (resp. P>o,±,t) be the orthogonal projection from the L 2 - 
completions of ^°°(<9M, (5±(TM) ®P)| 9Af) onto ©a^oPa,±,t (resp. ©a>oPa,±,t)- 

For any T G R, let (P+ T , P^o,+,t) (resp. (P^ T , P >0 _,t)) denote the corresponding 
Atiyah-Patodi-Singer boundary value problem [2]. More precisely, the boundary con- 
dition of D% T is P >0 , + , T {s\a M ) = for s G tf°°(M, S + (TM) ®P) (resp. of P^ T is 
P>Q,-,T(s|aMj = for s G ^°°(M, S^TM) ®E)). 

Both (P+ T , P^o,+,r) and (P^ T , P>o -,t) are elliptic, and (P^ T , P>o _,t) is the ad- 
joint of (Df T , P^ ,+,t) (cf- (11.81) . [SI Theorem 2.3]). Moreover, it is clear that both 
(P+ T , P^o,+,t) and (D^ T , P>o,-,t) are G-equivariant. 

Let Q% PS>T {E, ^) 7 G Z, 7 G G, be defined by 

(£, *) 7 " V? = Ind (P|, T , P>o, + ,t) 

(1.15) 7GG 

:= Ker (P^ T , P >0i+jT ) - Ker (P^ T , P >0 ,_, r ) . 
The following important result will be proved in the next two subsections. 

Proposition 1.1. For any 7 G G, there exists T 7 ^ such that Q^ PST (E, \1/) 7 does not 
depend on T ^ T 7 . 

1.3. An estimate on the boundary. The proof of Proposition II. II consists two steps. 
In the first step, we show that the boundary operator Dq MT) when restricted to the 
7-component, is invertible when T > is large. Then in the second step, we apply the 
spectral flow idea to complete the proof of Proposition 11.11 
In this subsection, we carry out the first step. 

For any 7 G G, let ^°°(<9M, (S(TM)®P)| aM ) 7 denote the 7-component of tf°°(dM, S(TM) 
®E)\ dM ). We denote the restriction of D§ MT on tf°°(dM, (S(TM)®P)| aM ) 7 by Pf M>T (7)@ 

Proposition 1.2. For any 7 G G, there exists T 7 ^ such that for any T ^ T 7 , 

Dq MT (j) is invertible. 



^Since Dg M T is G-equi variant, Dq MT [^) is well-defined. 
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Proof. Let g (thus g*) be equipped with an Ad^-invariant metric. Let hi, ■ ■ ■ , h dimG be 
an orthonormal basis of g* with its dual basis V{ of g. Then one has 

dimM 

(1.16) *(x) = *i( x ) V i, 

where 1 ^ % ^ dimG, are smooth functions on M. From (I1.16p . one gets 

dim M 

(1.17) * M (x) = E MWix). 

i=i 

Recall that from (TT7T2I) . ^ M \ dM e ^°°(dM,TdM). From (jTBjl . (TLlill and (jTTTjl . one 
finds, 

''dimM— 1 



(1-18) Wm,t) 2 -WJ 2 -^( E ^H 6dim ^ C (* M ) 



i=l 



-y^-i dim M— 1 



+ E c(e dimM ) C (e J )(Vf™^( C (e dimM )c(^))) 

i=i 

-v^Ttv| ( a T m) ^ + ^|v& m | 2 . 

For any 1 ^ i ^ dim G, let Ly. denote the Lie derivative of Vi acting on C £°°(M, S(TM)& 
E) and thus also on <af°°(dM, (5(TM) ® S)| aM ). Then 

V S(TM)®E _ ^ e ^oo (M; End(>S(rM) £)) 

is a bounded operator. 
By ( 11.171) . we write 

dimG dimG 

(1.19) v*iT w = E ^ + E *« (v*£ w - ^) • 

i=l i=l 

It is clear that when restricted on ^°°(<9M, (S(TM) ® i?)|aAf) 7 , each is bounded. 
On the other hand, since \I/ M is nowhere zero on dM, there exists C > such that 

(1.20) |^ M | 2 ^ AC on dM. 

From (|1.18p - (jl.20p . there exists a positive constant G 7 > such that for any s € 
^°°(«9M, (5(TM) ® £)M 7 , one has 

(1.21) 1 1 Dq MT s J | Q ^ ||^aM s ||a A f i0 — ^^7ll s llaAf,o + ^ 2( ^II' s I| 2 )m,o- 

From fll.2ip . one sees easily that Proposition 11.21 holds for T 7 = 2G 7 /G. □ 

1.4. Spectral flow and a proof of Proposition 11.11 It is an easy matter to extend 
the idea of spectral flow ([3l §7]) to the current 7-component situation. Recall that such 
an extension to the G-invariant case has already been considered in [221 §4a)]. 

Let {D t , O^t^ljbea one parameter smooth family of self-adjoint G-equivariant 
Dirac type operators acting on ^^(dM, (S + (TM)®E)\qm)- We define the 7-component 
spectral flow of {D t , ^ t ^ 1}, denoted by SP{D t , ^ t ^ 1}, to be the spectral 
flow of the family of self-adjoint Fredholm operators {£^(7), ^ t ^ 1} in the sense of 
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Atiyah-Patodi-Singer [3], §7], where for any t G [0,1], D t (j) is the restriction of D t on 
tf°°{dM,{S + (TM)®E)\ dM y. 

In view of ( 11.151) . one can then proceed as in [221 Theorem 4.2] (cf. also [6, Theorem 
1.1]) to get that for any T ^ T 7 , 

(1.22) Qf PStT (E, 9y - Qf PS ^ (E, *y = -SF {2?£,, +>t , T^t^T}. 
By (11.221) and Proposition ll.2[ we get Proposition 11.11 

Remark 1.3. Indeed, since the induced boundary operators are invertible for T ^ T 7 , 
the APS boundary problems involved form a continuous family of Fredholm operators, 
which implies the constancy of the associated index. By this one avoids the consideration 
of spectral flow in the proof of Proposition ll.il 

I. 5. Transversal index and the APS index. Denote by Qaps (-^j the quantiza- 
tion number Q% PS T (E, #) 7 for T > T 7 appearing in Proposition 11.11 

The following main result of this subsection identifies the transversal index in Section 

II. 11 with the above APS type index. 

Theorem 1.4. The following identity holds for 7 G G, 

(1.23) lnd,(a^)=Q A J PS (E^y. 

Proof. We can enlarge the manifold M a little bit to get a compact Spin c -manifold U 
with boundary such that M C U \ dU = U, the interior of U, and that everything 
extends in a G-invariant way to U from M, moreover, the corresponding vector field ^ u 
is nowhere zero on U \ M, with M the interior part of M. The existence of U is clear. 

Then by proceeding as in [5], §14], one can construct a complete metric g TU on U such 
that g TU \rM — 9™ , as well as an admissible (G-invariant) function / on U in the sense 
of Definition 2.6] such that f\u = 1, and / > ~ on U \ M, f — > +00 near dU. 

Let 

(1.24) D e Dt = D e + ^— /c(^) 

be the corresponding (G-equivariant) Dirac type operator on U. Then it is shown in [51 
§14] that when restricted to a fixed 7-component, for any T > 0, the restricted operator 
Dq t (t) is a Fredholm operator, moreover, one has 



;i.25) Ind (^| +)T (7)) = Ind 7 (a u E ^) 



where + ^(7) is the restriction of D~ T (y) on the 7-component of < t^'°°(U, S + (TU)®E) 
asin(rrTji 

As ty u is nowhere zero on U \ M, by the excision formula of the transversal index [TJ 
§3], 

(1.26) Ind 7 (<7*y = Ind 7 (a u E ^) . 

Now, the boundary dM cuts U into two manifolds with boundary dM : M and U\M. 
Let Df I--- (resp. Df 4 + T APS ) be the corresponding APS type operators ( i.e. the 
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APS type boundary value problems) on U \ M (resp. M) respectively as in Section [L2l 
Let ^fj^ + T APS {l) (resp. Df I + TAPS {^)) be the restriction onto the corresponding 
7- components. 

By proceeding as in [7J §3], which applies to the Fredholm operators here (one may 
deform the metrics and operators involved in a G-invariant way to the situation of 
product structure near dM, which does not alter the restriction of Z)~ on DM (thus it 
does not alter the indices considered), if necessary), one deduces that if T ^ T 7 so that 
Proposition 11.21 holds, then one has the following splitting formula, in which each index 
does not depend on T ^ T 7 , 

( L27 ) Ind { D l + ,T^)) = Illd ( D §\M, + ,T,AP S W) + Illd ( D M, + ,T,APS^)) ■ 

Lemma 1.5. The following identity holds for T ^ T 7 , 
(1-28) Ind (^ j+jTiAP5 (7)) = 0. 

Proof. Let U' be a G-invariant open subset of U\M such that U\U' is compact in U\M. 
Let U" be a G-invariant open subset of (U\M) U dM such that U'UU" = U\M and W 
is compact in U \ M. Let fi, f'2 be two G-invariant functions on U with supp(/i) C U', 
supp(/ 2 ) C U" such that fi, / 2 forms a partition of unity associated with the open 
covering U', U" of U \ M. The existence of f\, fi is clear. 

As the vector field is nowhere zero on U\M, and / > 1/2 on U\M, by proceeding 
as in [51 §9.3] and [221 §2] respectively, one knows that for any smooth section s lying 
in the 7-component of c to co {U\M 1 S(TU) ®E\q^) with compact support and verifying 
the APS boundary condition on dM as in Section 11.21 one has 

(1-29) H,tM[ > C (K(A s )llo + ( T " 6 )H^llo 

(1-30) H,tM[ > C ' (ll^(^)Ho + ( T ~ b ')Wh s Wl 

respectively, for some positive constants G, C, b' , and b. 

From (I1.29P and (11 .30[) and by taking T ^ T 7 large enough and proceeding the gluing 
arguments as in [H pp. 115-116], one gets Lemma [1.51 □ 

On the other hand, since }\m = 1, one sees directly, in view of Proposition 11.21 that 
when T ^ T 7 , 

(1-31) Ind (DfU^WT)) = Qaps (E, W . 

From fOBll -fTOB. we get (031 . □ 

Remark 1.6. Theorem II .41 allows us to use analytic method to deal with the transversal 
index problems. This is the viewpoint we adopt in this paper. 



1.6. A proof of Theorem 10. li We now apply Theorem 11.41 to the setting considered 
in Introduction. We have the canonical splitting TM ® M C = T (1 ' 0) M © T (0,1) M, for the 
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complexification of TM, with 

T (1 ' 0) M ={u G TM ® R C; Ju = V^lu], 

(1.32) . , 

T^M ={u G TM ® R C; Ju = -^f^lu}. 

Let T<°^M be the dual of T^M. For any a a 1-form on M, we denote by a* G TM 
its metric dual. 

The almost complex structure J on M determines a canonical Spin c -structure on TM 
with the associated Hermitian line bundle de^T^'^M) and we have 

(1.33) S{TM) = A (T* (0 ' 1} M) , S±{TM) = A^ (T<°^M) . 

For any W G TM, we write its complexification as W = w + w G T (1,0) M © T^M, 
let w* G T*^M be the metric dual of w (cf. H §5]). Then 

(1.34) c(W) = V2(uT A -i w ) 

defines the canonical Clifford action of W on A(T<°^M). It exchanges A evcn (T*(°' 1 )M) 
and A°dd( T *(o,i) M )_ 

The Levi-Civita connection V™ together with the almost complex structure J induces 
via projection a canonical Hermitian connection V T 1,0 M on T^'^M. This induces a 
Hermitian connection V det on det(T( 1,0 )M). The Clifford connection V A ( T ' >M ) on 
A (T*^ 0,1 'M) is induced by the Levi-Civita connection V™ and the connection V dct (cf. 
[21 §la)], [12, §1.3], OH Appendix D]). 

We take E = L and denote n°'"(M,L) = <af°°(M, A(T<°'^M) <g> L). We denote by 
L 2 (fi 0,, (M, L)) the L 2 -completion of the elements in Q 0, '(M, L) with compact support. 

Recall that we assume that the moment map fi : M — > q* is proper. For a regular 
value c>0of7i=|/i| 2 , denote by M c the G-invariant manifold with boundary 

(1.35) M c = {xE M : H{x) ^ c] . 

Also recall that X n is the Hamiltonian vector field of TC, i.e. ixnui = dJi. As in 
(11.16p . we can write 

dim G 

(1.36) y, = ^ Vih, 

i=l 

under the identification of q with q*, as 

dimG 

(1.37) fj. = ^ t^Vu 
and from M . one has (cf. [201 (1-19)]), 

dim G dim G 

(1.38) x w = -j{dny = -2 j ^ & (dm)* = 2 ^ ^ M = 2// M 

i=l i=l 

Since c > is a regular value of 7Y, by ( 11.381) one knows that X H = 2[i M is nowhere 
zero on dM c = 7^ _1 (c). 

We can now restate Theorem 10.11 as follows. 
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Theorem 1.7. For any 7 G G, there exists c 7 ^ such that for any regular values c', c 
of H. with c', c > Cy, the following identity for transversal indices holds, 

(1.39) Ind 7 (0%) = Ind 7 (a%) . 

7/7 = 0, we can take c 7 = 0. 

Proof. Let d > c > be two regular values of 7i. Let M C)C / denote the manifold with 
boundary M C)C / = M c / \ M c . 

By the additivity of the transversal index (cf. [U Theorem 3.7, §6] and [13, Prop. 
4.1]), one has 

(1 40) r 1 ( '•/ / \ W X r 1 ( '• I, . ' ' 



Ind 7 (<^)-Ind 7 «;)=Ind 7 



Thus ( I1.39P is equivalent to say that when c > is large enough, one has Ind 7 (o"^' c ') = 0, 
which by Theorem 11.41 is equivalent to say that 



(1.41) Qa c ps(L,^ = 0. 

If 7 = 0. by [221 Theorem 4.3], we get (Oil) for c' > c> 0. Thus we get (091 when 
7 = 0. 

From f)1.13p . we see that in the current situation, one has 



;i.42) 



Dif = D + 



r -\T 



(X n ) : ft ' (M c>cf , L) - ft -' (M c , c ,, L) . 



Let ei, • • ■ , edimAf be an orthonormal basis of TM CjC /. By [20, Theorem 1.6], one has 
the following Bochner type formula, 



;i.43) {Dtf=(D L ) 



>M 2 



-IT 



dim M 

E 



-IT 



-Tr 



T(!.0)M 



„ dimG 

?E 



Ml 2 



1=1 



dimG 



+ 4ttTW - 2v /z lT ^ ^Ly. + — \X 



8=1 



Let f2 0,, (M CjC /, L) 7 denote the 7-component of f2 0,, (M c c /, L). Then each Ly n 1 ^ i ^ 
dimG, acts on f2 0,, (M CjC /, L) 7 as a linear bounded operator A 7 (T^) and we denote its 
operator norm by ||A 7 (Vi)||. Set 

dim G dim G 

(1.44) F£ = (D^) 2 + 2 v /z TT ~ 2v /z TT ^ ^A, (^) . 

i=i i=i 

Lemma 1.8. There exists c 7 ^ such that for any c > c 7 , and any s G L 2 (fl 0, '(M, L)), 



supp(s) G M \ M c , one has 
(1.45) 



/dimG 



/ijA 7 (Vi) S, S 



^ {Hs, s) . 



i=i 



14 



XIAONAN MA AND WEIPING ZHANG 



Proof. By (11.371) . one verifies that 



;i.46) 



/dimG 



^ /ii^4 7 (Vi) s, s 



i=l 



^ dimG 

(IMlS+ll^rWHlS) 



2 

i=l 



dim G 



i=l 

>dimG 



From (11.461) . one then verifies easily that Lemma fL8l holds for c 7 = Y2i=i 11^7 (^») IP- 
Note that c 7 is bigger than the absolute value of the eigenvalue of the Casimir operator 
of G acting on the irreducible G-representation with highest weight 7, and c 7 = iff 
7 = 0. □ 

By (ll.43p - (ll.45p and proceeding in exactly the same way as in the proof of [201 Propo- 
sition 2.2], one verifies that the following result holds. 

Proposition 1.9. Let d > c > c 7 be two regular values of TC. Then for any x G 
M c ^> \ (dM c U dM c t), there exists an open neighborhood U x of x in M CjC > \ (dM c U dM c i) 
such that there exist C x > 0, b x > such that for any T ^ 1 and any s G Q°''(M,L) 
with supp(s) C U x , one has 



;i.47) Re ((F^s, s» ^ C x (\\D L s\\ 2 Q + (T 



On the other hand, since X H is nowhere zero on dM C)C i, the estimates described in 
Proposition 2.4] holds near <9M CC / for the 7-component here as well. By this and 
by Proposition 11.91 one can then proceed the gluing argument as in [22j Theorem 2.6], 
which goes back to [H pp. 115-117], to the the similar estimates for the 7-component 
(instead of the G-invariant component there), to see that the similar estimates in [221 
Theorem 2.6] still holds here on M CiC / for the 7-component. 

To be more precise, by Proposition II .21 one knows that there exists T 7 > such that 
for any T ^ T 7 , Dq M iT is invertible. 

From (11.211) for Dq M i t and (11.431) . and by the same argument as in the proof of [22} 
Proposition 2.4], we get : there exists an open neighborhood U of <9M CjC / and constants 
T > 0, C, d > such that for any T > T and s G Q°''(M CyC >, L) 7 , with supp(s) C U 
and P^o,±,Ts\dM c , = 0, the following inequality holds, 

(1.48) \\D^s\\l > c(\\D L s\\l + (T — COHsHg). 

On the other hand, by (11.441) one knows that for any s G f2 '*(M CjC /, L) 7 , one has 

(1.49) (D^) 2 s = F^s. 

By these and by the above discussions one sees that there exist C > 0, b > such that 
for any T ^ T 7 and any s G f2 0,, (M cc /, L) 7 such that P^o,±,T( s \dM ,) — 0, one has 

(1.50) \\D^\\l>C f (\\D L s\\ 2 Q + (T-b)\\s\ 



By taking T ^ T 7 sufficiently large in (11.501) . one gets (ll.4ip . which completes the 
proof of Theorem 11.71 and thus Theorem 10.11 □ 
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Remark 1.10. If the zero set of X n = 2/i M is compact, then Theorem 10. II is known in 
[17] and [25] already. 

Remark 1.11. Our proof of Theorem l0.4l in the following two sections follows the similar 
line of the arguments as in the above proof of Theorem 10.11 However, we have to deal 
with more subtle deformations and estimates, some of which are highly non-trivial. See 
the next two sections for more details. 

2. Quantization for proper moment maps: a proof of Theorem 10.41 

In this section, we prove Theorem 10.41 modulo a vanishing result Theorem 12.41 which 
will be proved in Section [31 

This section is organized as follows: In Section |2~T1 we give a reformulation of Theorem 
10.41 as Theorem 12.31 In Section 12.21 we state a vanishing result Theorem 12.41 away from 
# _1 (0). In Section [273| we give a proof of Theorem 12.31 by using Theorem 12.41 

2.1. A reformulation of Theorem 10.41 For convenience, we recall the basic setting. 

Let (M, uj), (N, uj n ) be two symplectic manifolds with symplectic forms uj, uj n respec- 
tively. We assume that N is compact. 

Let (L, h L ) be a Hermitian line bundle on M with Hermitian connection V L , and 
(F, h F ) a Hermitian line bundle on iV with Hermitian connection V F . Let R L = (V L ) 2 , 
R F = (V F ) 2 be the associated curvatures. 

We suppose that 



Let J M , J N be almost complex structures on TM, TN respectively such that J M ) 
defines a metric g™ on TM, and uj n (-, J •) defines a metric g™ on TN. 

Let G be a compact connected Lie group with Lie algebra g, and g admits an Ad^- 
invariant metric. Suppose that G acts (by left) on M, N and its actions on M, N lift 
on L and F respectively. Moreover, we assume that the G-action preserves the above 
metrics and connections on TM, TN, L, F and J M , J N . 

For any K G g, let K M G ^°°(M, TM) denote the vector field generated by K on M. 
Recall that the moment map [i : M — > g* has been defined in (10. 3p . Let r] : iV — >• g* be 
the moment map defined in the same way for (N, uj n ) and (F, h F , V F ). 

We will use the same notation for the natural extension of the objects on M , N to 
M x N. In particular, L <g> F is the Hermitian line bundle on M x N induced by L and 
F with the Hermitian connection induced by V L , V F . 

The G-action on M x N is defined by g ■ (x,y) = (gx,gy) for (x,y) G M x N. We 
define the symplectic form Q and the almost complex structure J on M x N by 



Recall that we have assumed that the moment map \i : M — > g* is proper. Since iV is 
compact, one sees that the moment map 9 is also proper. 



(2.1) 




(2.3) 



(2.2) 
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Recall that for any 7 G G, the index Q(L®F) 7 has been defined by applying Theorem 
10.11 on M x N, while Q((L <8> F)y) is the quantization number defined in [16] which by 
[TTl §7.4], can be defined as follows. 

If 7 is a regular value of the moment map 9, then one can construct the Marsden- 
Weinstein symplectic reduction ((M x N) 7 ,£Ly), where (M x iV) 7 = 6~ 1 {G • 7)/G is a 
compact orbifold. Moreover, L®F (resp. J) induces a prequantized line bundle (L®F) 7 
(resp. an almost complex structure J 7 ) over ((M x iV) 7 ,fi 7 ). One then constructs 
the associated Spin c -Dirac operator (twisted by (L (g> F) 7 ) on (M x iV) 7 whose index 
Q((L ® F) 7 ) as in (|Q2J is well-defined. 

If 7 G G is not a regular value of 9, then take a G g* sufficiently close to 7 such that 
a is a regular value of 9, and by replacing 7 by a, we get the index Q((L <S> F) a ). For a 
a regular value of 6* and close enough to 7, Q((L (8> F) a ) does not depend on a and we 
denote it as Q((L <8> F) 7 ). 

The following result can be viewed as a quantization formula for the 7 = (or G- 
invariant) component. 

Theorem 2.1. The following identity holds, 

(2-4) Q (L®F) 7=0 = Q ((L®F) 7=0 ) . 

Proof. If M is compact, this is the Guillemin-Sternberg conjecture proved in [15] and 

In the general case, for any two regular values d > c > of \9\ 2 , let Q APS c,c (1/ <S> 
F, ^) 7=0 be the APS type index defined in Section 11.51 for the current situation, where 
(M x N) cy = {xe M x N :c^ \9(x)\ 2 ^ c'}. Then by [22], Theorem 4.3], one finds 

(2.5) QZ X s NUc '(L®F,9y=° = 0. 

By Theorems 10.11 11.41 (11.401) and (12.51) . and by taking d > large enough, one sees 
that for any regular value c > of \9\ 2 , one has 

(2.6) Ind 7 = (4^; )C ) = Ind 7=o (Cy) = . 

where (M x JV) C = {{x,y) e M x N : \9\ 2 ^ c}, and similarly for (M x A^) c /. 

If is a regular value of 9, then from [22] Theorem 4.3], Q { Ap x s N)c {L <g> F, 6 I ) 7=0 = 
Q r(L®F) 7=0 V From Theorem O] (j2T6]l . we get (jUJ) . 

On the other hand, if we take c > small enough so that there is no critical point 
of \9\ 2 besides 9~ 1 (0), then by [17] Prop. 7.10] and [18] one knows that even if is a 
singular value of 9, 

(2-7) Ind 7=0 = Q ((L®F) J=Q ) . 

From ( 12.61) and ( 12.7ft , one gets (12 .41) which completes the proof of Theorem 12.11 □ 
Remark 2.2. See Remark 12.81 for an outline of an analytic proof of (12.71) . 

By Theorem 12.11 one can reformulate Theorem 10.41 as follows. 
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Theorem 2.3. The following identity holds, 

(2.8) Q (L®F) 7=0 = Q( L V Q (F)- 1 . 

■yeG 

In the rest of this section, we will present a proof of Theorem 12.31 modulo a vanishing 
result away from _1 (O), which will be stated in the next subsection and proved in the 
next section. 

2.2. A vanishing result away from # _1 (0). In this section, we state a vanishing result 
which can be thought of the first step in the two steps deformation proof of Theorem 



We use the Ad^-invariant metric on g to identify g and g*. Let Vi, 1 ^ i ^ dimG, be 
an orthonormal basis of g, then we can write the moment maps fi and r\ as 

dim G dim G 

(2.9) 0=2 HiVi, V=Y1 

i=l i=l 

For any 1 ^ i ^ dimG, denote by V^ M , V/^ , "\Z MxAr the Killing vector fields on 
M, N,M x A induced by Vi respectively. Then one verifies easily from (10.41) that 



I 1 

N \ rN 



(2 ' 10) (d M my =j m v™ , (^)* = j^ 

Let D mF : tt°'*(M x iV, L cg> F) -> fi°'*(M x A, L g> F) be the Spin c Dirac operator 
onMxjV (cf. Section Ol (jOjl ). 

By Sard's theorem, the set of critical values of the functions |/i| 2 and \9\ 2 on M x iV 
has measure zero in R. Especially, for any C > 0, there exists C > C such that C" is a 
regular value for the functions |/i| 2 and ||#| 2 on M x AT. 

As A is a compact manifold, there exists Cq > such that |r/| ^ Co on A. 

For A > 6Cq large enough which is a regular value of the functions |/i| 2 and ||#| 2 on 
M x A, we define 

A4 ={(x,y) eMxjV; |/i(x)| 2 ^ A, \6(x,y)\ 2 < 2A} c M x A, 

M A ={s G M; |/i(x)| 2 < A}, 8M A = {x G M; ^(x)] 2 = A}, 

(2 ' H) A4i ={(x,y) G M x A; |/i(x)| 2 = A}, 

Al 2 ={(z,y) G M x A; |#(x,?/)| 2 = 2A}. 

By our choice of A, we know that M\, Al 2 are smooth sub- manifolds of M x A. More- 



over, as 

|2 I ,, |2 i i |2 



(2.12) \6\ 2 = \fi\ 2 + \t]\ 2 + 2(^r]} 

and A > 6Cq, we know that M\ H Ai 2 = 00 Thus Ai is a smooth manifold with 
boundary dM. and 

(2.13) &M=AliUAl 2 , A4i = 8Ma x A. 



7 Indeed, by (12T2D . one gets ^ \0\-C . Thus, if |6f = 2A, then |^| > AV^^-l^/a-Ch) > 



A 1 / 2 when A > 6Cq. 
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We will introduce a deformation of the vector field Y^=i ^j^Vj^ + Vj*) on .Mi to 
the vector field Ylf^[ G Qj^Vj* + Vj*) on -^2, where 9j = fij + rjj, j — 1, - • • , dimG, are 
the coordinates of 9 associated to the basis Vj, j — 1, • • • , dimG, of g. 

Let if>j, j = 1, • • • , dimG, be smooth functions on M such that there exists a strictly 
positive function ip G ^^(pM) such that 



(2.14) ^(x,y) 



<p(x,y) Hj{x,y) on A^i, 
(p(x,y)9j(x,y) on M 2 - 
Let X p be the vector field on M defined by 



dim G 



(2.15) X"= X 0i W • V H- 

3=1 

As I 1 2 and \9\ 2 are G-invariant functions on M x N, Mi and A^ 2 are G-submanifolds 
with boundary of M x AT, in particular, 

(2.16) Vf IxN {x,y) G T&M, X p (x,?/) G TdM, for G &M. 

As A is a regular value of the functions \fi\ 2 and h\9\ 2 on M x N, we know that 



X p = J ^ £?=T W + Vf) + pointwise over A4i, 
" \p J2*=i G 83 (Vj M + Vj N ) pointwise over M 2 . 

Thus X p is nowhere zero on dM. 

Clearly, X p is induced on M by the G-equivariant map X p : M — > g defined by 

dim G 

(2.18) X»=J2^ V r 

We can now state the main result of this subsection as follows. 

Theorem 2.4. When A > is large enough, there exist functions ipj, 1 ^ j ^ dimG, 
verifying the above properties, such that the following identity for the APS type index 
holds, 

(2. 19) Qf PS (L ® F, X p ^j 7 ~° = 0. 
The proof of Theorem 12.41 will be given in the next section. 

2.3. A proof of Theorem 12.31 We continue the discussion in the previous subsection. 

Denote hy ^ = M A xN = {(x,y) G M x N : |/i(a;)| 2 ^ A}, Ji 2 = {(x,y) G M x N : 
^(x)! 2 ^ 2A}. Then j^ 2 are submanifolds with boundary oi M x N such that 
= M h i = 1,2. Recall also J{ 2 = ^1 U Al 

Let ?/> : ^2 -* S be a G-equivariant map such that the induced vector field ip~^ 2 on 
^#2 verifies ?/>"^ 2 |.m = X p . The existence of t/> is clear. 

From (12.171) . the positivity of <p on dM and the additivity of the transversal index (cf. 
[H Theorem 3.7, §6] and [TFl Prop. 4.1]), one has, 

(2.20) Ind 7=0 = Ind 7=0 (o^,) + Ind 7=0 (af^) . 
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From Theorems 11.41 12.41 and (12.201) , one gets 
(2-21) Ind 7=0 (*fi F J = Ind 7=0 «^) . 

Lemma 2.5. The following identities hold 

^ Illd 7=0 (<rf®F,4>) = Ind 7=0 «§F,J > 

Ind 7=0 (crf£ F ^) = Ind 7=0 (vf® Ffi ) . 

Proof. For any t G [0, 1], set fa = (1 - t)ip + tfa 6 t = (1 - *)V> + t0. Let /.if 1 , df 2 be 
the induced vector fields on respectively. Then by (I2.17P and the positivity of 

ip, one sees that fa 1 (resp. 6^) is nowhere zero on cL#i = .Mi (resp. cb#2 = M-2)- 
Formula (I2.22p then follows from the homotopy invariance of the transversal index (cf. 
[D Theorems 2.6, 3.7] and [17, §3]). □ 

Recall that by our assumption, the induced vector field fi M of fi : M — > g is nowhere 
zero on DM a. 

Proposition 2.6. TTie following identity holds, 

(2.23) Ind 7=0 = E Ind ^ Ki) Q W ■ 

Proof. Proposition [22] can be proved by using the homotopy invariance of the transversal 
index. Here, we will develop an analytic proof. 

Recall that fi MxN denotes the vector field generated by [i on M x N. One has 

dimG 

(2.24) M MxW = £ fa (Vf + Vf ) . 

i=i 

For any £ G [0, 1], we define the deformation 

dim G 

(2.25) ^m = T,^( v i M+ o--t)v j N ). 

For any t G [0, 1], T ^ 0, we define the following Dirac type operators as in ( 11.131) . 

D!f F (t) =D L ® F + ^y^c {^(t)) 
(2-26) : L ® F) -> fi '*^, L® F), 

Dlf(t) =^1^^. 

Let (D^rf (t), P^o,+,T(t)) denote the canonically associated (elliptic) APS boundary 
value problem defined similarly as in SectionO Let md 7=0 (£>+^ F (£), i^o,+,r(*)) denote 
the G-invariant part of the corresponding index. 

Lemma 2.7. There exists T > sttc/i i/iai /or any T ^ T , Ind 7=0 (-D+ ( j^(t), P^o,+,t(*)) 
does not depend on t G [0, 1]. 
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Proof. For any T ^ 0, t G [0,1], let Dg®f T (t) be the Dirac type operator on the boundary 
dJt x induced by D^ F (t) (cf. (Oil ). ' 

In view of Section 11.41 in order to prove Lemma 12.71 we need only to show that 
there exists T > such that when restricted to the subspace of G-invariant sections of 

n°>* 

On Mi, set 



i, L <g> F)\ d ^, D L a % T {t) is invertible for any T ^ T , t G [0, 1]. 



(2.27) 



X(f) 



\fJL 



MxN\2 



MxN 



Y{t)=^{t)-X{t). 



Then X(t), Y(t) are perpendicular to each other. Moreover, one verifies that the follow- 
ing identity holds on A4i, 



(2.28) 



EdimG xrM i /1 jA v^aimLi t t 



dimG 



Edim G t t 
i=i ViVi 



+ 



Edim G t r 
i=i Mi^i 



1/2 



^ c A > 



for some positive constant ca which might depend on A. 
Now we write Dg®f i T (t) explicitly as, in view of (11.141) . 



(2.29) D L d % F M)=D L d % 



IT 



p>L®F 

U dJ( 1 



c{e dimM )c(^{t)) 
~ — c(e dimM )c(Y(t)) 



-IT 



c(e dimM )c(X(t)) . 



Then as X(t), Y(t) G TcL#l are perpendicular to each other, one has 

\/-LT 



(2-30) (D L 9 X, T (t)Y 



[D%%,c(e dimM )c(X(t))} 



+ 7 l\X(t)\ 2 +(D d ^ 



-IT 



c(e dimM )c(Y(t)) 



By ( 12.281) and proceeding similarly as in the proof of Proposition 11.21 especially (11.181) . 
( I1.19P , one gets that there exist T > and C > such that for any T ^ T , t G [0, 1] 
and any G-invariant section s of fi 0,, (^#i, L (8) T)^^, one has 



f2.3i: 



- [^af, c (e dimM ) c (X(t))] + I \X(t)\ 2 ) s, s) > C(s, s). 



From ( 12.311) . one sees that for any T ^ T , i G [0,1], when restricted to the subspace of 
G-invariant sections of f2 0,, («/#i, LigF)^^, Dg^ T (t) is invertible, thus (T>^|f (t), P^o,+,t) 
form a continuous family of Fredholm operators, which implies the constancy of the as- 
sociated index. The proof of Lemma [2.71 is thus completed. □ 

Coming back to the proof of Proposition 12.61 Let /i Mj4 = Ylt=l G ^Nt^ be the vector 
field on M A induced by jj, : M — > q. By ( I2.25p . one has 



dimG 



(2.32) 



M A 



on 



8=1 
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Now we deform inside Ma (leaving the data on 8Ma unchanged) in G-invariant manner 
the metrics and connections as well as \i Ma to the situation that everything is of product 
nature near 8Ma- We denote also the spinor S{TMa) of Ma (associated to the product 
metric near the boundary now) obtained from A(T*( 0,1 ^M). Then when taking product 
with N, we also deform things on to a situation which is of product nature near 
dJt x =Mi = 8M A x N. 

We then attach an infinite cylinder 8Ma x [0, +00) along the boundary 8Ma and ex- 
tend everything in Ma to the now complete manifold Ma with cylindrical end, and \i Ma 
is constant along the direction [0, +00). By taking product with N, we get similar con- 
struction for M\. We denote the extended subjects on the obtained manifolds with 
cylindrical end by a "~" modifying notation. Then S(TJ^) = S(TM A ) <8> A(T* (0,1) iv). 

Now since for T ^ To, Dg^ T (l) is invertible on the subspace of G-invariant subspace, 
the standard arguments in (2J Prop. 3.11] show that 



(2.33) ' - 



: L 2 S+iTJKx) ®L®F\ ~^L 2 Lg u S^TJKi) ® L ® fY 

where we use the superscript G to denote the subspace of G-invariant sections, is a 
Fredholm operator. Moreover, its index equals to Ind 7= o(-D+^ F (l), P^o,+,t(1))- 

Let ei, • • • , edimM (resp. /1, • • • , fdimN) denote the orthonormal basis of TM4 (resp. 
TN). Then we can write 

dim M 1 — jvj-, dim N 



(2.34) D^ F = c( ei )V s J T ^® l ® F + ^-^c L M A + c (/i) 

i=i 3=1 

where £2?* c(e,) vff^^+^c^) (resp. c (fj) vf^~^ F ) is lifted 

from a corresponding operator on Ma (resp. iV) denoted by Dif (resp. D F ). 
By ( 12T34D . one has 

(2.35) (bl® F ) 2 = (pfj\(D F )\ 
from which one gets 

(2.36) Ker L 2 (r)|® F ) = Ker L 2 ® Ker (L> F ) . 

Since AT is compact, dim(Ker.D F ) is of finite dimension. From (12.331) . (I2.36P and the 
standard arguments in |2, Prop. 3.11], one then deduces that when T > is large 
enough, 

(2.37) Ind 7=0 (^/(1),P^, + ,t(1)) = Y.^s^^f ■ Q{F)-\ 

By taking A > large enough, from Theorem 11.41 Lemma [2.71 and (12.371) . one gets 
(I2.23p . which completes the proof of Proposition 12.61 □ 

From (I2.21I) - (I2.23I) and by taking A > large enough, in view of Theorems 11.41 and 
11.71 one gets (12.81) which completes the proof of Theorem 12.31 
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The proof of Theorem 10.41 is thus also completed. 

Remark 2.8. By combining the argument in this subsection with those of [2TJ and |22j . 
one is able to get a new proof of (12.71) . Indeed, if c > is a regular value of |#| 2 , for a G Q* 
close enough to G g* which is a regular value for 9, let O a be the co-adjoint orbit of a 
and consider the moment map 9 a : (M x N) c x O a — > g* defined by 9 a (x,y) = 9(x) — y 
for x G (M x iV) c and y G O a . Then as \a\ is small enough, d\9t a \ 2 , t G [0, 1], are nowhere 
zero on d(M x iV) c x O a . One can then use the argument in this subsection to relate the 
G-invariant APS type index on (M x N) c to that on (M x N) c x O a , which in turn leads 
to the contribution on 9~ 1 (0)/G = 9~ 1 (Ga)/G, by combining the arguments in |22j and 

mi, 

3. Vanishing result: a proof of Theorem 12.41 

In this section, we will establish our vanishing result, Theorem 12.41 Assume that X p is 
a suitable vector field on Ai (depend on the parameter A) in (12.151) deforming /i Mx7V on 
Mi to 9 MxN on M 2 , we need to prove that when T > is large enough, the G-invariant 
component of the APS type index of the deformed Dirac operator D^ 1 in (13.91) on Ai is 
zero. In fact, we will prove that after we fix A large enough, the restriction of Dj? with 
APS boundary condition on the G-invariant sections is invertible for T large enough. 

As X p is nowhere zero on dAi, by the argument of (11.481) (cf. [221 Prop. 2.4]), we 
have an estimate (13.1171) for Dj? on an open neighborhood of dAi similar to (jl.48p . By 
the gluing argument as in |U p. 115-116], if we establish the local estimate (I3.94p inside 
Ai , then we can conclude Theorem 12.41 

As we are interested in the restriction of D T ' 2 to the G-invariant subspace of Q 0, '(A4, L® 
F), we will take the term ipjL Vj in (13. lip as zero. If z £ zero(X p ) = {z G Ai, X p (z) = 0}, 
then from the Bochner type formula (13. lip , the term ^-|A P | 2 is the leading term and we 
get easily (I3.94H . If z G zero(A p ), then we hope to adapt the argument in [20, §2b] to 
get the local estimate around z. Basically, we hope ipj9j will be a positive term, and it 
will control all the tensors in (13.111) . As there is no assumption on the vector fields Vj M 
and their covariant derivatives on M, we can not expect to get our estimate by a simple 
argument. To control locally the covariant derivatives of V^ 1 in (13. lip near z G zero(AT p ), 
we will use the harmonic oscillator argument from [20l §2b)], thus we like to impose that 
the vector filed ipjVj^ is a Hamiltonian vector field along the direction M. 

As N is compact, the tensor and their derivatives are bounded. Thus if on 
zero(X p ), we can control \J— l(Ji + -Z2) which involve V^ 1 and (d M ipj)* in (I3.10p by 
ipj9j, then basically, we can achieve our local estimate. What we prove in Proposition 
13.111 (whose proof occupies from Section [3731 until Section [3771) is that for our choice ipj in 
(13.41) . v 7 — 1(^1+^2) is bounded from below uniformly on A > A , any regular value of \fi\ 2 
and \\9\ 2 , and on zero(X p ) C Ai. Moreover ipj9j ~ |/i| 2 ^ A on Ai, thus when A > 
is large enough, the term 27rijjj9j will really control the tensors which do not involve the 
derivative of Vj^ in (13. lip . In this way, we can obtain the local estimate (I3.94p around 
each z G zero(A p ). Now from the gluing argument of [H p. 115-116], (I3.94p and (I3.117p . 
we get (13. 118p . which imply the invertibility of the restriction of the operator D^ 1 with 
APS boundary condition to the G-invariant part. Especially we establish Theorem 12.41 
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This section is organized as follows. In Section I3.1[ we propose a construction of the 
deformed vector field X p in ( 12.151) which depends on two functions a, <fi. In Section 
we establish a Bochner type formula A3. 1 1 j) for the Dirac operator deformed by X p . 



In Section I3.3[ we study the relation on the zero set of X p in M. of the vector fields 
generated by the group action, and in Section 13.41 we study the asymptotics of the 
functions appeared in the above relations when the parameter A — > +oo. In Section 
13.51 we compute precisely the tensors involving the vector field Vj M and (d M i/jj)* on the 
zero set of X p in the Bochner formula (13. lip . In Section [331 we study the coefficients 
appeared in the computation in Section [331 In Section [3771 we prove in Proposition ^. Ill 
that the sum of the tensors involving the vector field Vj M and (d M if)j)* on the zero set 
of X p in the Bochner formula is uniformly bounded from below for our choice ipj. In 
Section [331 under the help of Prop. 13.111 for A large enough fixed, we establish the local 
estimate around each point inside M.. Finally in Section [331 we prove Theorem 12.41 

We use the notation as in Section El and when a subscript index appears two times in 
a formula, we sum up with this index. 

3.1. The construction of the deformed vector field. We first specify the deforma- 
tion vector field used in Section 12.21 
Let 5, 4> e *f°°(R) be such that 

5 (t) =( t2 ' mA 1 '^ fort< r 

(3.1) \l, fort^f, (2(1 -t), for t>|, 

~ 29 ~ 12 

a{t)+(f>{t) > — , <f>'(t)<0, for -^t^ -. 

The existence of 5, is easy to see. For example, one may set 5o(t) = t 2 , (j)o(t) = 1 — t 3 
on t ^ |; a (t) = 1, <j>o(t) = 2(1 — t) on t ^ |; and a , (f> are linear on | ^ t ^ |. By 
an approximation argument from 5?q, 0o, one gets 5, verifying ( 13. lft . 

For A > 0, set 

(3.2) a(0 = 0W = ?(^- 
Let /3 G #°°(M x AT), p:MxiV-4g*~gbe defined by 

(3.3) = |/i| 2 + a (|/i| 2 ) (|fl| 2 - M 2 ) , p = 9 - <P((3)V- 

Recall that Vi, • ■ ■ , VdimG is an orthonormal basis of q, V^ 1 , are the Killing vector 
fields on M, N induced by V,. For any function Q with values in g, we will denote Qi its 
i-component with respect to the base {Vi}. Sei@ 

7i =2 (1 + ol (|/i| 2 ) (|^| 2 - l^l 2 )) H + 2«(|/i| 2 )^, 
V>j = 2 Pi - W(P)PiVaj- 
Let X p be the vector field on .M in (12. lip defined by 

(3.5) x p = ^ (y/ + vf) . 



See Lemma 13.71 for a possible motivation of introducing the ipj 's here. 
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Then one verifies, 

X p =2 N {V 3 M + V 3 N ) on .Mi, 

(3 ' 6) x p = (2 + 2m)oj(vj t + v/ f ) on ^ 



We can now state the main result of this section, which precise Theorem [231 as follows. 

Theorem 3.1. When A > large enough is a regular value for both \fi\ 2 and ||#| 2 , X p 
is nowhere zero on Mi and Mi- Moreover, we have 

(3.7) g^ s (L®F,ir° = 0. 

We devote the rest of this section to the proof of Theorem 13.11 

3.2. A Bochner type formula for the deformation by X p . We will fix the sym- 
plectic form uj m and the almost complex structure J M on M induced from M x N . 
Then the associated Riemannian metric g™ is also induced by g™ and g™ . Let V™ 
be the Levi-Civita connection on (M, g™). 

For any function ip on M x N, we denote by d M ip, d N ip its differential along M, N 
respectively. 

We denote also Vf 4 the Killing vector field on M induced by V? e Q, then 

(3.8) V 3 M = V 3 MxN = V 3 M + Vf . 

Let V T<1, ° )M , V T(1 ' 0)7V be the connections on T^M, T^N induced by the Levi- 
Civita connections V™, V T7V as explained in Section 11.61 Let V A °'* be the Hermit- 
ian connection on A(T*( 0,1 \M x N)) <S> L F induced by the Clifford connections on 
A(T*(°' 1 )M) and on A(T<°'^N) and the connections V L , V F . 

Definition 3.2. For Tel, let Djf 1 be the operator defined by 

(3.9) = D mF + ^^c(X p ) : fi°'"(«M, L ® F) -> fi '*(A^, L ® F). 

Recall that when a subscript index appears two times in a formula, we sum up with 
this index. Set 

h =\c ((d"^)*) c ft" + 2*f ) + ic (W) c ft") , 



(3-io) /2= K( i+ ^)^ M ' 



h=\c{{d N ^Y)c{V 3 N ). 

Let {e/c}^™ M (resp. {fi}i=l ) be an orthonormal frame of TM (resp. TN), then 
{ea}^™^ := {e^} U is an orthonormal frame of TM. 
The following Bochner type formula holds for (Djf 1 ) 2 . 
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IT 



dim M 



(3.11) D^ = D L ^ + ^- £ c(e fc )c(V™(^^)) 



fc=i 



dim N 



/ —1T 



Tr 



i=l 



-IT 



-ifjj Tr 



r7 T( 1 '°)N-\rN\ 

T 2 



+ 2ttT^ - yf-lT^Lv. + ^|X"| 2 + v/^TT (Ji + I 2 + J 3 ) . 



Proof. For any 1 ^ a ^ dim.M, it is clear that 

(3.12) Vf C (X0 = c(X>) Vf + c (V™X>) . 

By (EU), (13TT21 . we have 

(3.13) 



D M,2 = D L®F,2 + 



-IT 



dimA4 

E (c(e a )Vf C (X")+ C (^) C (e fl )Vf) + ^Xf 

a=l 



2 



dim 



T 2 



E c ^ c (VZ M * P ) - v^TTVi ; + L - \x 



a=l 



By (201 Lemma 1.5], 
(3.14) V^m = L v . + 2^^^, 



-. dim A4 



a=l 



Thus by (1331) . ([331, 



(3.15) v^T = Vi-ky* + iH-Hfa 

1 dim A/ . dimiV 

+ E ^)c(V«V/')ft + i £ c(/,)c(V™l/")^ 



fe=l 



+ Tr 



i=l 



V | T (l,0) M 



+ Tr 



r-rTP^jVT/JVl 

V | T (i,o) jy 



Thus from ([33]), flSTTOj) . fl3TT4j) and fl3TT5|) . we get 



dim M 



(3.16) Vf P ' = ^L Vj + 2nV^l^ +1 E c ( e *) c ( V ™ fe^ M )) 

fc=i 

1 dimiV 



i=l 



+ 



j ) It(!>o)m 



+ Tr 



Jv y. 
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Also by fl33|) . 
(3.17) 

1 dimM , dimM 

\ E <<«J° W x ') = 5 E <<*)<= ( v ™ + 5c ((d"*n c W) 

o=l fe=l 

dim AT 



2 ^ wv v /! 2 

i=l 

By f l3TT0D . flHZED, (BUHD and f[3T7D . the proof of Theorem E3J is completed. □ 

3.3. Relations between terms in (13.111 ) near zero(X p ): Part 1. In this subsec- 
tion, as well as the next a few subsections, we establish certain formulas concerning the 
relationships between the terms appearing in (13.111) . on the zero set of X p in A4. These 
relations are crucial for the proof of Theorem 13.11 
Set 

T 1= l + a'(H 2 )(|C-H 2 ), 
r 2 =1 - 2(f)'((3)p i r) i T 1 , 

(3.18) r 4 =1 - <P(0) - 2 ( j ) '(f3)a(\fi\ 2 ) Pl r ]l , 
r 5 =(l-0(/3))r 1 -a(|/.| 2 ) 

=1 - m - «(i/ii 2 ) + (i - mwwm 2 - m 2 )- 

These functions appear naturally on zero(X p ), as is clear from the following (13.191) 
and Lemma [3.41 

From (HOD, (EHHD, one gets 

(3.19) 7j =2r 1 ^ + 2a(| / u| 2 )^. 

From fl33D, fl3TT8|) and fl3TT9|) . one sees 

2 ^ = 2/i, + 2(1 - <j>(f3)) Vj - 2<p'((3)p iVi (2n^- + 2a(\p\ 2 ) Vj ) 

= 2r 2 /ij + 2r 4 7/j. 

Lemma 3.4. On zero(X p ) = {a; 6 A1 : X p (x) = 0}, we have 

(3.21) r m Vf l = - r m V™, r 2lj V™ = -2r 5 r^ M , 
and 

(3.22) r^Vf = - nrjjVf, r^Vf = -2r 5 ^-Vf , r^Vf = (r 2 - r^Vf . 
Proof. By (12.151) . one sees that 

(3.23) X p = if and only if ^^ M = and ^^^ = 0. 

j j 

From ( 13.20p . the equation ^2jif)jVj* = in (13.231) is equivalent to the first equation 
of (I3T2T]) . 
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By (I3.18p . (13.191) and the first equation of (I3.2ip . we have on zero(X p ), 

(3.24) r 2lj Vf = 2r 1 r 2 p 3 V J M + 2a(\p\ 2 )r 2 r ]j V J M 

= -2 [(1 - m - 2<P\{3)a(\p\ 2 )p i r ]l ) n - a(\p\ 2 ) (1 - 2<j!(p)pwr x )] rjjV™ 

= -2 ((1 - ~ a(H 2 )) rjjV? = -2t^V 3 m . 

In the same way, by f)3.20p . J2j 4>jVj N = in (13.23}) is equivalent to the first equation 
of (13T22D . 

By (12. 31) and the first equation of ( 13.221) . as in ( 13.241) . we have 

WiV? =2 Wj -Vf + 2a(\p\ 2 )r 2Vj Vf = -2 nVj Vf , 
r 2 j V 1 N =r 2 p 3 Vf + r 2Vj Vf = (r 2 - r,) Vj Vf. 
The proof of Lemma 13.41 is completed. □ 

3.4. Estimates of Tj, i = 1,2,4,5, when A > is large. Recall that {3, <p have been 
defined in (E2D, (13T31) . 

Lemma 3.5. There exists A > such that for A > A , we have 

(3.26) A < (3 < 2A, on M \ dM. 
Thus 

(3.27) < (f)((3) < 1 on M \ dM. 
Moreover, uniformly on M and the parameter A, we have 

(3.28) ri = l + (D(A~ 1/2 ), r 2 = 1 + 0(A-^ 2 ), 

(3.29) n = (l-m)a + 0(A^ 2 )), 
(3.30 ) 75 = (1 - 0(/3) - a(M 2 )) (1 + 0{A- 1 ' 2 )). 
Finally, for any A > A , we have 

1 -4>{0) -a(\p\ 2 ) < if {x,y) e M\dM, 

= if(x,y)edM. 

Proof. As iV is compact, there exists Co > such that 

(3.32) \r]\ <: C . 

From f[2T[j) . (I2TT21 and (l3T32|) . one sees that on M, one has 

2A^ |/i| 2 + \ V \ 2 + 2(p, V ) 
(3-33) .i x 9 

\ I 1 2 i I 1 2 ( L \ |2 i o| |2\ \ ^ I |2 o^<2 

^ M +\m -{^w + 3 M ) > 3H -2C . 

From (I3.33p . one knows that when A > is large enough, one has on M that 

(3.34) A 1/2 ^ \p\ < V^5A 1/2 . 

From O, (|2TTT|) . (EH])-([3l3D and (J33|), one deduces that for A > large enough, 
one also has 

(3.35) \6\ < 2A 1 / 2 , \6\ 2 - \p\ 2 = 0{A 1 ' 2 ), (3 = \p\ 2 + 0(A 1/2 ) on M. 



(3.31) 
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If \fx\ 2 < 1.3A, then by (I3.35p . when A > is large enough, one has (3 < Then by 
(EUD-dSISD, one has for (3 < M 



3 > 



4-i 



12 



(3.36) A V ,^ 2 



i)(i + i(M!-i)W-l,P 



= (^-i)(i + 0(^ 1/2 )). 

If |/i| 2 ^ 1.3A, then by (13351) . one has 

(3.37) f3 ^ A+ (0.3A + O(A 1/2 )) > 1.2A, 

when A > is large enough. 

From dSHD, fl33J, dS3D, (HD2D, dS3SD and (13371) . one gets (13361) and (13371) . 
From (12~TTD . (1331) . (1331) and (13321) . one gets 

(3.38) |p| < 2A 1/2 on M 
From (J33J, f[3~TBj) . (13321 . (1335]) and (13351) . we get (13351) . 

On the other hand, if |/i| 2 < 1.3 A and A > is large enough so that /3 < then by 
( 1331) and ([33]), one has 

w(/?H( r; )3 ' . 

From (1331) . (1331) . (13321) . (13381) and (13391) . one finds 

0'(/5)«(H 2 )=(l-0(/5)) 4/3 O(^ 1 ), 
0"(/5)a(|/i| 2 )p^=(l-0(/?))O(A- 3 / 2 ). 

By ( 1332j) . (13381) and (133D]) . we see that (13391) holds for A > large enough and 
H 2 < 1.3A 

While if |/i| 2 ^ 1.3A, by (1337]) . when A > is large enough, /3 > 1.1 A, and thus 
by (J33J and (l33j) . one has 1 - ^ 1 - 0(1. 1A) > 0, from which (1339]) holds 

tautologically. 

The proof of (13.291) is thus completed. 

If |/i| 2 > ^, then from (1331) . 

(3.41) a(|Ai| 2 ) = l,a'(|A*| 2 ) = a*(|A*r) = ) 
from which, one gets 

(3.42) l-0(/?)-a(|/i| 2 ) = -0(/3), r 5 = -0(/?). 

Thus from (13351) and (13321) . we get (1330]) and (133T|) . 

Now by (13. ip . (13.21) . (13.31) and (I3.35p . it is clear that when A > is large enough, 

(3.43) m = <PW)+<P'W)0{A 1 / 2 ). 
Let < eo < | be such that 

(3.44) 5 (t)+J(f)^, f or iz^° <t< | 
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By (13. ip . the existence of e is clear. 

If ^=|eM <; ^1 2 ^ M ; then from (jO) . (Ig^j) and (13^3j) . one sees that for A large 
enough, 

(3.45) 1 - - a (H 2 ) ^-1 

thus we get again (13.301) and ( I3.3ip from (13.181) . 

If |/z| 2 < ( 4 ~^°) A ; then by (I3.35p . one sees that when A > is large enough, one has 
(3 < ^. Thus, by (13TT1) . (15^ . (13351) and ([339]) . one has 

a'(M 2 )(l - = " l) ((^f " + O^ 1 / 2 ))) 3 OiA- 1 ) 

(3.46) = (M!_ 1 y 0(A -i ) , 

1 - m - a{\rf) = -(M! - l) 2 (l - (M! - l)(l + O(A-V^))). 

from which and (13.181) we get again (13.301) and (13.311) . 

The proof of (13.301) and (I3.3ip is now complete, as well as Lemma 13.51 □ 



The following Lemma will also be used in the proof of Proposition 13.111 
Lemma 3.6. There exists A > such that for any A > A Q , 

,3.47, l< ",-ff-g < U °" M ^ 9M - 

1 - <j>{P) - a(\fi\ 2 ) 

Proof. In fact, by (l3T2Tj) and fl3~3T|) . 

(3.48) (1 - 0(/3)) 2 - a{\fi\ 2 ) < 1 - 0(/?) - a{\fi\ 2 ) <0onM\ dM. 
Thus we need to prove that 

(3.49) 11 - 1O0O3) - lla(M 2 ) - 0(/3) 2 < if (x, y) e M \ dM. 

If |/i| 2 > then a(|/i| 2 ) = 1, thus (ICTj) holds. 

Let e > be defined as in (13.441) . 

If A < |/i| 2 < ^f^, then by (KT21) . (13T36D and (13391) 

(3.50) 11 - lO0(/3) - lla(|/i| 2 ) -0(/?) 2 

f-') 2 (-^ 12 (f')-(ff^"))' 

From (I3.50p . we know (13.491) holds for ^4 large enough. 

If ^ |/i| 2 ^ then by (13T45|) . as < </>(/3) < 1, we get, when A > is large 

enough, that 

(3.51) 11 - lO0(/3) - ll«(|/x| 2 ) - 0(/3) 2 < -H + 0(/3) - 0(/3) 2 < -1. 

Thus (jS3H) holds on A4 \ □ 
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3.5. Relations between terms in ( 13.11ft near zero(X p ): Part 2. The first of two 
relations we would include in this subsections is 

Lemma 3.7. We have 

(3.52) ^V™ = -J M (d M \p\ 2 )*. 

Proof. By (J23J), (E3J), we have 

d/3 =2(l + a'{\p\ 2 ){\6\ 2 - \p\ 2 ) - a(M 2 ) + 2a(\n\ 2 )6 j d6 j 

(3-53) =ljd M fij + 2a(\p\ 2 )6 j d N 7 lj} 

d Pj =d9j - <j)'(/3)r}jdf3 - (f){l3)d N r] j . 

From (jgJUJ) and fl33Bl . we get 

(d0)* =J M Vf + J N V 3 N , 



(dpy = lk j M v k M + 2a(\^\ 2 )e k j N v k N , 

(3 ' 54) =j m v m - 0'(/%- 7fc j M < 

+ (1 - m)J N Vf - 2<p'{{3)a{\p\ 2 )r lj 6 k J N V k N . 



From (E3D, fl334D . we get 
(3.55) = 2p 3 [V 3 M - <j>'{P) m ikV k M ) = -J M (d M \p\ 2 y. 

Thus we get (13321) . 



□ 



On the other hand, by Lemma 13.51 we see that when A > is large enough, r 2 > 1/2. 
To state the second relation we would include in this subsection, we introduce the 
following two functions, 



n(3)a"(\p\ 2 )p lVl (\0\ 2 -\^\ 



4<p'((3)a'(\p\ 2 ) PtVt T4 



T-2 



T 5 



(3.56) 



b'(P)a'(\n\ 2 )pir}i 



[T2 ~ T A )T 4 



20"(/3)a(| / z| 2 )p^ + 2(0'(/3)) 2 a(|/z| 2 )^ 



2a 2\ l T 2 - T 4)T5 



+ W) 



(t 2 - r 4 )r 5 



+ l-20(/3) p-a(M 



t- 5 



^2-74 

r 2 



which appear naturally in the following lemma. 
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Lemma 3.8. On zero(X p ) C Ai, we have 

(3.57) \c{{d M ^r)c{V 3 M ) + j ( (l + V™, (d^i)*) 
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(c(J M ^)c(^) + ^L=|^| 2 ) 



+ /4- |c(%J M Vf)c(^^) + 



and 



(3.58) 



±c((d^)*)c(lf ) =\r 2 ■ c(J M Vf)c(Vf) + h ■ c^V^c^V"), 
)c((d N ^r)c(V^ =\r, ■ c(J N Vf)c(V 3 M ) + h ■ c( Vk J N V k N )c( Vj V 3 M ). 



(3.59) 



4 U r j ' ' y j 2 
Proof. By Q, d33D, we have 

d 7i = [2c/'(M 2 )(|0| 2 - |/i| 2 )/i, - 2c/(M 2 )/^ + 2a'(l^| 2 )^] 

+ Aa'{\fi\ 2 )^B k dO k + (2 + 2c/(|/i| 2 )(|fl| 2 - M 2 ))d%- + 2a(|/i| 2 )rf%. 

From fl2TT0|) . (13391 . we get 

(d 7j -)* =4a"(|/i| 2 )(|0| 2 - \^ m J M V k M 
+ 4a'(\»\ 2 )( W + r ]kt , j )J M V k M 

+ (2 + 2a/{\tf)(\0\ 2 -\tf))j M V j M 



(3.60) 



+ Aa\\p\ 2 )p 3 6 k J N V k N + 2a(|/x| 2 ) J^V/. 



Now from Q, (T3T54|) and fl3~60|) . we get 
(3.61) (d M Vi)* = -2(^/3)^7, (d M /?)* - 2<f/{p)-q i {p i {d M lj y + 7j {d M ' Pi )* 

+ 2{d M Pj y 

= -2<j>"(P)p imj l k J M V k M - 2^'(P)p lVl Ua"(\p\ 2 )(\9\ 2 - \p\ 2 )^ k J M V k M 



+ 4a'(\rf)(v j »k + VkP j )J M V k M + [2 + 2a/{\tf)(\0\'-\tf))J M Vf 
- 2cf>'{(3)r ]ilj {j M V l M - mmkJ M V k M ) +2(J M V 3 M - mWkJ M V k M ). 
Thus from (KIM and $3M) , 

(3.62) (d M ^)* = 2t 2 J m V 3 m - 8^(P)a"(\p\ 2 )pM\0\ 2 ~ \fi\ 2 )^ k J M V k M 

- 80'(/V(H>^(w + Vk»j)J M v k M 

+ 2(- 4t'iP)p<m + ^'m 2 rihlkJ M V k M - 24>'{(5){r ]klj + r) jlk ) J M V k M . 
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By (EH]), 

(3.63) c{{d M ^)*)c{V 3 M ) = 2r 2 c{J M V 3 M )c(V™) 



- tyWCM'W c{p k J M V k M )c{r, 3 V 3 M ) + c{r, 3 J M V M )c^ k Vl 



+ 2( - 0"(/3)p^ + (0 , (/5))\ 2 Jc( 7fc J A V fc M )c(7 J ^ 

- 20'(/3) \c( Vj J M V 3 M )c( lk V h M ) + c( lk J M V k M )c(v 3 V 3 M ) 



and by using the fact that J M is anti-symmetric, ( ) on TM is C-bilinear, we get 



(3.64) 



J 



M 



V", (d M ^Y) ) = 2r 2 



M|2 



- S^^a'd^l 2 )^-^ <r^, p fc Vf > 
+ 2( - 0"(/3)p^ + (0'(/?))\ 2 ) -L=| £ 7 *^ 



-24>'((3)^=( Vj V j M ,p k V k M ) 



For the term c((d M V' j )*)c(\/ iV ), from fl3TT8|) . (13T621 . we have 

(3.65) c((d M ^)*)c(^) = 2r 2 • c( J M Vf)c(Vf) 

- %<P\f3)a'\\rf)pM\0\ 2 ~ \p?)c{p k J M V k M )c{ N V 3 N ) 
- 80W(M 2 )P^ [c(/i fc J M V fc M )c(^K w ) + c( Vj J M V 3 M )c(p k V k N ) 



- 0"(/3)p^ + (0 , (/?))\ 2 Jc( 7fc J M n A/ )c(7,^ 

- 2^09) [c{ m J M V 3 M )c{l k V k N ) + cWVftcfaV") 
Moreover, from fl2TT0l) . (EMj) , (EM and pi = 6 { - <p{(3)r]i, we get 
(3.66) = -20 / (/5)p i r /i (rf%)* - 2<j> , {(3)r Hlj {d N PiT 



- 2<j>"{P)p i r hl j(d N PY ~ 2cj ) , (f3)pa j (d"r h y + 2{d N Pj ) 
= -20(j3) m [Aa\\p\ 2 )p 6 k J N V k N + 2a{\p\ 2 )J N V 3 N 

- 2<p'(P) vn3 [(1 - m)J N V t N - 2<p'((3)a(\p\ 2 ) Vl 9 k J N V k N 

- 4<p"(p)a(\p\ 2 )p lV aA-J N Vk N - ty'tfW - mvih r J N v t N 



+ 2 



(1 - m) J N Vf - 2<J ) '(P)a(\p\ 2 )r lj 6 k J N V k N 
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From (gUD and f[3~66]) . 

(3.67) (d N ^T = 2r 4 J N V 3 N - 8^K(H 2 WA^X^ 

+ ( - 40" (/^(M 2 )^ + 4(0'(/?)) 2 a(|/i| 2 )r /i 2 - 20'(/?)) 7 ,^, J N V k N 

From f)3.67p . one has 

(3.68) c((^)*)c(\// f ) = 2r 4 ■ c( J^Vf )c(\A M ) 

-8 ( p\P)a'(\p\ 2 )p i r ]i c(9 k J N V k N )c(p 3 V J M ) 

+ ( - 40"(/3)a(|/z| 2 )p^ + 4(0'(/3)) 2 a(M 2 H 2 - 2<j>\pj)c(9 k J N V k N )c( 7j V^) 

- 2^(/J)(l - 2^)c( Vi J I, V i rf )c( lj Vf i ) 

-40'(/3)«(|/i| 2 )c(^J Jv ^)c(r ? ,^). 

On zeropf), from (137211 (137631) and (137EID . we then get 



(3.69) \c{{d M ^Y)c{Vn + + V j 



r 2 - ( c (j M ^)c(y/) + -^=|^ 



I n ilf n\ II l\ ,.|2\ „ „ /|/1|2 I , |2\ /' T 4\ 2 



4 [ Vr 2 
+ 8^a'(M 2 ) Pl ^ + 2( - n/3) PiVi + (0'(/?))\ 2 ) (^) 2 

+20'(/?)-^} ■ (c{v k J u V?)c{r, s V?) + ^L=| E^r) 



From Q37MJ) we get (137oTj) . 

On zeropf), from (1372Tj) . (13722]) and (13765]) . as in (13769]) . we get the first equation of 

(GLSED. 

On zero(X^), from (13721D . (13~22l) and (13~68l) . we get 
(3.70) \c{{d N ^y)c{V 3 M ) = ir 4 • c^V^ciV™) 



40"(/3)a(|/i| 2 )p^ + 4(0'(/3)) 2 a(M 2 )r / 2 - 20'(/?) 



2(r 2 - r 4 )r 5 



r 2 

'2 



+ 20'(/?)(l - 20(/?))^ -40'(/5)a(|/i| 2 )^^) c( Vk J N V k N )c( Vj V^). 
From (I3.70p . we get the second equation of (I3.58p . □ 
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3.6. Estimates of J 4 and I 5 . In this section, we establish the following estimate result 
for the terms J 4 , I5 appeared in Subsection 13.51 

Lemma 3.9. When A > is large enough, we have 

h =2<f>'(i3) (l - m - a(M 2 )) (1 + 0(A~ 1/2 )), 

h =W)((1 - m? ~ «(H 2 ))(1 + 0{A^ 2 )). 



(3.71) 



Moreover 
(3.72) 



h>0 if (x, y) e M\ dM, 

h = if (x, y) e dM = M x u M 2 - 



Proof. If |/i| 2 ^ ^, by Lemma [331 (Q, (jSiSSD , (I3T36D and fl3738|) . we get 
(3.73) J 4 = ^(flO^Xl - </>(/3)) 2 - <P'(P)0(A^ 2 )(1 - 0(/3)) 

+ 2( - 0"(/?)p^ + (0'(/3))\ 2 ) (l - 0(/3) - a(|/x| 2 )) 2 (l + 0{A-W)) 

+ 2^09) (l - 0(/3) - a(|/i| 2 )) (1 + O^- 1 / 2 )). 

By (EU), (ET351) . ([CTD , fl3~45l) and (l3~46l) . there exist C > such that for any A > A , 
if |//| 2 ^ M then 

1 - ^ C\l - <f>(J3) - a(|//| 2 )|, a(\^\ 2 ) ^ C\l - <j>(J3) - a{\tf)\, 

|^(/5)|^C|l-0(/3)-«(|/i| 2 )|. 

By (13T39|) . fl3~731 and (13321) . we get fl33Tj) for J 4 . 
If |/x| 2 > ^, then by (13"3)) . fl3~4TD . fl3~42]) and (133561) . 

/ h = o(A^ 2 )m 2 - mewm + ^(^ 1/2 )) 

(3.75) 

= -2<j ) '(mm + o(A- 1 / 2 )). 

Thus we get the first equation of (13.711) . 

As 4>'{t) < for t > 0, thus <J/(J3) < on |/i| 2 > A, from f[333Tj) . (13331) . we get (1333]) . 

If |/i| 2 ^ ^, by ([31]), (15351) . for A large enough, 0(/5) ^ 0(^) + 0(A~ 1/2 ) > §, and 
so by dS2HD and (133291) . 

(3.76) r 2 -r 4 = 0(/3) + O(A- 1 / 2 ). 

Thus if |/i| 2 ^, by Lemma EH (jBTT]) . (15381) . (153UD and f|3776l) . we have 

(3.77) J 5 = 0'(/3)(l - 0(/?))O(A" 1 / 2 ) + (1 - «/>(/?)) (l - 003) - a(H 2 ))0(A- 3 / 2 ) 

+ <j>'{(5) (l - 0(/3) + OiA- 1 ' 2 )) (l - 0(/3) - a{\^\ 2 )) 

Now by (13391) . (1538]) . (1534j) and (13~77j) . we get (153TD for J 5 . 
If |/x| 2 > ^, then by (15381 and (153T]) . we have n = 1 and 

(3.78) r 2 -r 4 = 0(/5). 
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Thus if |/i| 2 > ^, by (ETIJ) . (13381 . (l374Tj) . f[3^2j) . f[3~56l) and ([3T781 . we get 

(3.79) 7 S = ^mm^fOiA 1 ' 2 ) + 0'(/3) [ - 0(/3) 2 (l + 0(,4~ 1/2 )) 

- (1 - 20(/3))0(/3)(l + O^- 1 / 2 )) - 0GS)(1 + 0{A~ l/2 )) 



As < 0(/3) < 1 on M \ dM, from and fl3T9l . we get if |/i| 2 > 
(3.80) h = -0'(/5)0(/5)(2 - 0(/3))(l + 0(A~ 1/2 )). 



5A 
3 ' 



From ([SaiD, flBTBDD . we get again flBTTTj) for J 5 . 

The proof of Lemma 13.91 is completed. □ 

3.7. Estimates of I\+I 2 - In this subsection, we establish an estimate result on zero(X p ). 
Before doing this, we first prove the following lemma. 

For any x G M, y G N, W G T X M, let B(W) G End(A(T*(°' 1 )(M x W))) (sr , y ) be defined 

by 

(3.81) B(W) = y/-[c(J M W)c(W) + \W\ 2 . 

Lemma 3.10. For any W G T X M, V G T y N, the endomorphisms B(W), y/-Lc(W)c(V) 
of A(T* (0,1) (M x N))( X) y) are self-adjoint and for any k > 0, 

B(W) > 0, 

(3.82) t i 

V=lc{W)c(V) > T B(W)-k\V\ 2 . 

2k 

Proof. As W, V are orthogonal to each other, B(W), V '—lc(W)c(V) are self-adjoint. 

For W G T X M, V G T y N, we write their complexification as W = w + w, V = v + v, 
with w G T^M, w G T^M, v G T^N, v G T^N. Let W* G T< ^M, v* G 
2 1 *(o,i) tli e metric duals of w, v. Then we deduce from f ll.34p that 

B(W) = c(-w + w)c(w + w) + \W\ 2 

(3.83) 

= -2(w* A +i w )(w* A -i w ) + 2|w| 2 = Aw* A i w . 

Thus we get the first equation of (13.821) which has been obtained in [201 (2.9), (2.13)]. 
Moreover, 

(3.84) \/-ic{W)c{V) = 2 V /Z T (w* Av* A -w* A % - i w v* A +i w %) . 
If a G A(T*(°' 1 )(M x N)) (x , y) , then we can write 

(3.85) a = w* A v* A o x + w* A a 2 + v* A a 3 + er 4 , 
and <7j (i = 1, 2, 3, 4) do not contain the terms w* or v*. Then we get 

B{W)a =4\w\ 2 w* A v* A a x + A\w\ 2 w* A a 2 , 

(3.86) >/ = Ic(W)c(V> = - 2y/^T\w\ 2 \v\ 2 a 1 + 2^/^^\w\ 2 v* A a 2 

- 2 v /Z T|f \ 2 w* Aa 3 + 2\f-Lw* Av* A a 4 . 

From (137861) . 

(3.87) (B(W)a,a) = 4|w|>| 2 | ( x 1 | 2 + 4|w| 4 |cr 2 | 2 = 4\t w a\ 2 . 
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From ([3T86]) . <^M) , we get for any k > 0, 



(3.88) 



-lc(W)c(V)a, a) = A\w\ 2 \v\ 2 Im(cri, <x 4 ) - 4|w;| 2 |w| 2 Im(cr 2 , <x 3 ) 



2 1 1 2 



From (13.871) . (13.881) . we get the second equation of ( 13.821) . 



□ 



Recall that the terms I\, I2 have been defined in (13.101) . We now state the crucial 
estimate for I\ + I2 as follows. 

Proposition 3.11. There exist C > 0, Aq > such that for any A > Aq and (x,y) G 
zero(X p ) C M., we have 

(3.89) v c l(/i + / 2 ) > -CId. 

Proof. By ( 13.101) . and apply Lemma [3. 101 to (13.571) and (I3.58P with k = 8, we get 

/ -. \ dim G dim G 

v^i(/i + / a ) ^ u - s 7 " 2 " 8 T4 E B{ y^ - ( 8r2 + 8r4 ) E i*f i a 

V / i= i i= i 



(3.90) 



16/4 + 16/5 



dim G 



E ^ 

3=1 



TV 



By Lemma I3"3| < </>((3) < 1 on M. \ dAi, we get for A large enough 



(3.91) 



T~2 - -T"2 - -T4 



+ h(P) + 0(A- 1 ' 2 ) > 0. 



From Lemmas 13.61 13. 9[ we know that when A > is large enough 
(3.92) 



J 4 - I 1 * ~ I 1 * >lh>0 if (x,y) G M \ (M 1 UM 2 ). 
000 



As I VI- I, 1 77 1 have upper bound not depending on x G M, thus \ V, 



N I 



EdimG t r 
3=1 VjVj 



N 



are uniformly bounded on Ai. 

As zero(A^) C M\dM, from (I3T82D . (13301) . (13~MD and (gQBD , there exist A > 0, 
C > such that for any A > A , and (x, y) G zero(X p ) C X, (I3T891) holds. □ 

3.8. A local point estimate around zero(A p ). Recall that Djf 1 is the deformed Dirac 
operator on M. defined by ( 13.91) according to the functions defined in ( 13. ip . ( 13. 2ft and 
( I3T3D . For any T ^ 0, let Ffi 1 : ft°''(.M, L®F)^ W(M, L ® F) be defined by 



(3.93) 



The purpose of this subsection is to prove the following local pointwise estimate around 
each x G zero(A p ). 
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Proposition 3.12. There exists A > such that for any A ^ A a regular value of 
both |/i| 2 and j\0\ 2 , and that if M. is constructed as in h2.11\) , then for any z G M. \ dM., 
there exist an open neighborhood U z of z and C z > 0, b z > such that for any s G 
f2 0,, (.A/l, L <g> F) with supp(s) C U z and T ^ 1, one has 

(3.94) Re ((F^s, s)) > C z (H^^H* + (T — b z ) || S || 2 ) . 

Proof. Recall that {ft} is an orthonormal frame of TN. Set 



(3.95) Rj 



dim TV 



Tr 



jT^Nt/N 



8=1 



By ( jSHJ) , (13331) and (13T95D . 



(3.96) = D mi " ;2 + 



fc=i 

J-1T 



Tr 



[v T(1 ' 0)M (^. 



J^ M )It(!>o)m 



T 2 



+ T (2ttV^ + ^ + V^l(h + h + h)) + ~j\X p \ 2 . 

By (ISTTOD . fl3TT8|) . (j3TT9"j) . (13T35D . (ISToTj) . # = /i + r/andp = #- we know that 

there exists C > such that for any A large enough, on A4 



(3.97) 



^ C, thus \F\ ^ C. 



Recall that by (13T20]) . ^ = 2r 2 /i,- + 2r 4 ^. Thus from Lemma EH (El, (12TT2D . (13321) 
and 



l/2l 



(3.96 



|Vil = 0(i4 

= 2r 2 \fi\ 2 + 2r 4 M 2 + 2(r 2 + r 4 )(/i,^) 
= 2| y u| 2 (l + C(A- 1 / 2 )). 



By Proposition EHH (I3T2]) . (13T38]) . (13371) and (13T98D . there exists A > 0, such that 
for any A > A which is a regular values of the functions |/i| 2 and ||#| 2 on M x iV, and 
(x, G {X p = 0} C M, we have 



(3.99) 



2 ^ j e 3 + -^-(^ + it;*) + V^i(h + i 2 + j 3 ) > A, 



Let's fix z = (x , y ) G M\ dM. 

If XP{x ,y ) + 0, then (13~M1) follows trivially from lEEBBjl . 
If X p (xq,?/o) — 0, we will adapt the argument in [201 §2b)]. 
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Let ex, • • • , edimAf be an orthogonal frame of TM. Set 

dim M 



(3.100) 



A M = E ( K y_ v ^ 

3=1 

dim JV 

A7V = E ((< *) 2 - v vf ' AMxJV = AM + AJV ' 



be the Bochner Laplacians acting on ft°''(Af, L), fi°''(iV, F), fi°>*(M x N,L®F). From 
the Lichnerowicz formula for ]J L ® F < 2 [T2"| Theorem 1.3.5], [TTj Appendix D], one finds on 
M, 



(3.101) 



D 



L®F,2 



-A 



MxN 



0(1). 



Let ei, • • • ,edimM be an orthogonal base of T X0 M. Let (xi, ■ • • , £dimM) be the nor- 
mal coordinate system with respect to {ej}^ 1 ™^ 1 near xq. Certainly, we can choose 
ei, • - • , edimM such that the function |p| 2 (-, yo) has the following expression near xq, 

dimM 

(3.102) \p\ 2 (x, y ) = \p\ 2 (x , y ) + ^ a,x| + 0(\x\ 3 ), 

3=1 

where the a/s may possibly be zero. 

The following Lemma is an analogue of [2"0"1 Lemma 2.3]. 

Lemma 3.13. The following inequality holds at the point (xo,y ), 
(3.103) 



■ dim M 



fc=i 



V 



T(!.o)M/ 



(V^ m )It(m) M 



dim M 

^ - E N 

3=1 



where the inequality is strict if at least one of the aj 's is negative. 

Proof. Let e"i, • • • , edimM be an orthonormal frame of TM near x so that at x , ej = ej 
for 1 ^ j ^ dimM. From Lemma [3.71 and (I3.102p . we find that 

dim M 

(3.104) (V^/%,3/o) = -2 tjfayo^ej, and t 3 {x,y ) = a jXj + 0{\x\ 2 ). 

3=1 

From (ETglj) . (I3T83D . (13.1041) . we deduce that at the point (x ,y ), 

■ dim M 



1 



E c(e,)e(V™(^f))-^Tr 



4 



i^v 3 M )\ T , mM 



k=l 
■ dim Af 



j aim in > — t- dim A/ / /• \ 

- g a,c(e,)c(J M e,) E \ ( X + T^J H* J 

^ dimAf ,,,,, .,, 

= 9 E "lWes)-2)>- E 1° 



(3.105) 



dim Af 



where the last inequality is strict if at least one of the a/s is negative. 
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Let (yi, ■ ■ ■ , ydimiv) be a normal coordinate system near y on U yo C N. Let F^ 4 * be 
formal adjoint of F^ 1 . From §M>, (Ell), (jSZEjlD , (l3~103l and fj3TT04l) . we find that 
near (x ,l/o), 



dim M 



dim Af 



(3.106) -(F^ 4 + F^ 4 *) ^ -A MxN + T ' ^( x ^o) 2 

j=i i=i 

rp2 dimG 2 

+ tI E^l +TA + 0(1 + T\x\). 

3=1 

Following [2U1 §2b)], let e > 0, which will be further fixed later, be sufficiently small 
so that the orthonormal frame {ej}^™ M is well defined over 

(3.107) B e (x ) = {xe M; d(x, x ) < e}. 
Let (V ej )* be the formal adjoint of V^ '* on B e (xo). Set 

dimM 

(3.108) -Ay = ((V ej .)* + r( S gn % )t,(a;,2/o)) +T(sgna j )t J -(x,y )) ■ 

Clearly, — is nonnegative when acting on compactly supported sections over B e (xo). 
Moreover, we verify from (13.1041) that 

dim M dim M 

(3.109) _A^ = -A M -T l^jl+T 2 ^ ^-(x, y ) 2 + 0[d x + 1 + T\x\), 

i=i i=i 

where by 0(d x + 1 + T\x\) we mean a first order differential operator 



dim M 



d 



(3.110) bj(x)— + d(x) with ^(x) = 0(1), cf(z) = 0(1 + T\x\). 

j=l o 

We will also use similar notation for other operators. 

From (13.1061) . (13.1081) . (13. 109j) . we get, when acting on sections with compact support 
in B £ (x ) x U yo , for any k > 1, 

-{F™ + Ffi 4 *) ^ -A N - A? +TA + 0(d x + 1 + T\x\) 



(3.111) 



dim M 



> ~A N - \a m -1 J2 \ aj \+TA+ 0(d x + 1 + T\x\). 

K K K 

3=1 



For any s G fl°'*(M,L <g> F) with supp(s) C B e (x ) x C/^, by (13.1011) . it is standard 
that there exist Ci,C 2 ,C 3 > such that 



(3.112) 



-{A MxN s, s) ^ C^D^sWi - C 2 \\s\\l 



(0(1 + T\x\)s,s) <C 3 (l + Te)|H|g. 
Also by the elliptic estimate and Cauchy inequality, there exist C 4 , C5 > such that 
(3.113) (0(d x )s,s) ^C,e\\D L ® F s\\l + ^\\s\\l. 
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From Hill - fl3TTT3|) . we get 



dim M 



\s\\o 



-t 1VJ, 

(3.114) Re ((F^s, s)) > (~f ~ Qe) ll^^^sl^ + t(A - C 3 e - - ^ |a,- 

Cs C's \.. ||2 
T + T + C 3 j|| S || . 

Now, we take k large enough such that A — ^ X]j=i A/ l a il > ^4/2, then we choose £ small 
enough such that 

(3.115) -±-C 4 e>0, A/2 - C 3 e > A/A. 

k 

With this choice of e, by (I3.114|) - (13.1151) . we get Proposition 13.121 at (x ,y ). □ 

3.9. Proof of Theorem 13.11 We now assume that A ^ A is a regular value for both 
|/i| 2 and ^\0\ 2 , with Aq verifying the conditions in Proposition 13.121 In particular, by 
(13.61) . X p is nowhere zero on dM = Mi U M 2 - 

Let e„ be the inward pointing unit normal at any boundary point of M. Set 

\f—\T 

(3 n6) ^ =D™ - ^c(e n )c(I") 

: fi°''(A*,L®F)| aA , -+QP> m (M,L®F)\ gM 

be the Dirac type operator on dM induced by Djf 1 . 

Since X p is nowhere zero on dM, in view of (13.51) . by Proposition 11.21 we know that 
when restricted to the G- invariant subspace of fl°'°(M,L eg) F)\q M , there exists T\ ^ 
such that for any T ^ T 1; Dgj^ ^ is invertible. Moreover, a similar formula like (11.211) 
holds. 

One can then proceed as in the proof of (11.481) . to see that there exists an open 
neighborhood Uqm of M and positive constants C\ > 0, b x > such that for T ^ T x and 
any G-invariant element of Q°'°(M, L®F) such that supp(s) C Uqm and P^o,±,t(s\sm) — 
0, where P^o,±,r are the APS projections associated to Dqm,t (cf. Section fl~2l . one has 

(3.117) ||^ s |lo > C i (ll^^llo + ( T " 6 i) II s ! 

From (I3.93p . (13.1171) and Proposition 13.121 one can then proceed as in the proof of 
(ll.50p to see that there exist C > 0, b > such that for any T ^ T\ and G-invariant 
element of Q°''(M, L ® F) such that -P^o,±,t( s |sm) = O5 one has 

(3.118) ll^llo > C (]\ DmF 4l + ( T - b ) No) • 
By taking T ^ T\ large enough, we get Theorem 13 . 11 
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